Python: fuzzy searching

In building this patch management application, I was having a hard time trying several ways to compare a dictionary of the names of packages to another of the name of software titles.

A little bit of searching and I learn about a Python library called ‘FuzzyWuzzy’ that uses ‘Levenshtein distance’ to measure the metric distance between two strings.

It now means that I post a software title and find all the packages that match within a metric that I set.

The basic layout is below with a metric match of 90%

from fuzzywuzzy import fuzz

def search(values, searchFor):
    # Get the values form dict
    for k, v in values.items():
        # make the value a string (some were int)
        v = str(v)
        # Skip empty packages
        if v == 'None':
            pass
        else:
            # lowercase the str to increase match
            Partial_Ratio = fuzz.partial_ratio(searchFor.lower(), v.lower())
            # over 90% and good to go
            if Partial_Ratio > 90:
                return v
            else:
                pass


def main():
    # Function that gets package id & names from API
    pkgs = get_all_packages()

    # Function that does same for software titles
    sw_titles = get_all_software_titles()

    # Loop through the sw titles
    for sw in sw_titles:

        # assign the name to var
        sw_name = sw['name']

        # print for testing
        print('SW Title: ' + sw_name)

        # Loop through packages
        for pkg in pkgs:

            match = search(pkg, sw_name)

            # print if theres a match
            if match is not None:
                print('Match: ' + match)

Using macOSLAPS and retrieving the local admin account from Active Directory

After a laptop theft at my office, I am implementing Joshua Millers macOSLAPS.

The laptop had turned up in the Netherlands after a guy had got in touch with us wehn he had carried out an internet recovery. Because it was still under our DEP account, it pulled down our configuration and so the logon screen had our branding!

Now it’s all very well that our machine is still on DEP but it raised the question why we still had LAP and firmware passwords that were ages old and known by everyone and their dog. So going forward and to be compliant a bit better with security, I had to come up with a solution to rotate the passwords but have them on record.

So implementing macOSLAPS should be fairly straight forward, you’d think but I have some legacy scripts that use a local admin account as well as newer ones such as creating a securetoken for a user as we image the machine before handing it to the user.

Therefore how do you access the local admin password?

It’s kept in 2 places when using macOSLAPS: Active Directory computerobject under ms-Mcs-AdmPwd and in the System keychain

It is neigh on impossible to get it from the System keychain as the machine presents a GUI to enter the admin password so it cannot be done programmatically.

So that leaves accessing Active Directory but the problems I encountered were as follows:

  1. Using dscl presented an error regarding the DC URL when using credentials but then couldn’t access the attribute without them
  2. Trying through Python and the LDAPS3 modulewas problematic (mostly because I’m such a Python novice!)

So it came down to trying to find another way to do this through tools embedded into macOS and accessed through Terminal.

It came down to this:

  1. Use security to get the machines domain password
  2. Use that to carry out an ldapsearch to find the computer object
  3. grep the results to get the ms-Mcs-AdmPwd:
  4. Write this all into a function to easily add into Shell scripts
  5. Go on a search through every single script to find any instance where the local admin credentials were required (mostly securetoken and filevault 2 ones)
# Gets the hostname
cName="$(hostname)"

# Gets the domain
domain="$(dscl localhost -list "/Active Directory)"

# Do a clever bit here to split the domain into dc= bits

# Gets the machine domain password
dcPwd="$(security find-generic-password -w -s "/Active Directory/$(dscl localhost -list "/Active Directory")" /Library/Keychains/System.keychain)"

# Gets the LAPS attribute from Active Directory
laps="$(ldapsearch -H ldaps://dc.company.tld -x -W -D "$cName" -b "dc=domain,dc=name" "(cn=$cName)" | grep ms-Mcs-AdmPwd: | awk '{print $2}')"

As for the firmware password, I’m planning out a Python script that will have the following steps:

  • Generate an alphanumeric string of 12 characters
  • Save it to a file locally & hash the contents (so we have previous pwds for any eventuality)
  • Attempt to update the FW password
  • If successful, write to the JSS API into an extension attribute

List Comprehension

Learnt this from someone today. Instead of writing a full for... loop

goodlist = []
for x in mylist:
  if x['package'] != None:
    goodlist.append(x)

It can be shortened to…

goodlist = []
goodlist = [x for x in mylist if x['package'] != None]

Test SMTP from macOS Terminal

  1. telnet host-to-test 25 (connect to port 25 on mail server)
  2. HELO sending-host
  3. MAIL FROM: foo@foo.com
  4. RCPT TO: bar@bar.com
  5. DATA
  6. (enter one blank line after DATA)
  7. Subject: test
    To: to-user
    From: from-user
    (enter one blank line after From:)
    test text for email
    . (enter a single period by itself on the last line)
  8. QUIT

Drive Space is almost full

Kept getting this message on my work MBP:

Then found that there was a folder called ‘cores’ that was full of 278GB of dumps!

Once I deleted them on Terminal they kept reappearing so had to use this command:

sudo launchctl limit core 0 0
sudo sysctl -w kern.coredump=0

But this won’t be persisted through a restart. To do so create a new file on /etc/sysctl.conf (if it does not exist yet) and write in the following configuration:

kern.coredump=0

Uploading Patch Definitions to Patch Server

I wrote a quick Python script to upload patch definitions hosted the external patch server by Bryson Tyrell for use with Jamf Pro’s patch management.

It’s a simple process of reading a directory, opening each file, loading the JSON then carrying out a requests to POST the data through the API

#!/usr/local/bin/python3

import requests
import json
from pathlib import Path

dirpath = 'path/to/patchdefinitions.json'
url="patchserver.url"
headers={"Content-Type": "application/json",}


def send_request(file_contents):
    try:
        response = requests.post(url = url,
            headers = headers,
            data = json.dumps(file_contents))
        print('Response HTTP Status Code: {status_code}'.format(
            status_code=response.status_code))
        print('Response HTTP Response Body: {content}'.format(
            content=response.content))
    except requests.exceptions.RequestException:
        print('HTTP Request failed')


def read_file(dirpath):
    pathlist = Path(dirpath).glob('*.json')
    for path in pathlist:
        # because path is object not string
        path_in_str = str(path)

        print(path_in_str, "\n")
        
        with open(path_in_str, 'r') as jf:
            file_contents = json.load(jf)

            send_request(file_contents)
            jf.close()

read_file(dirpath)

More Things I Keep Forgetting!

[This is a work in progress and will be added to as I go!]

I have been using Docker to containerise some small services that don’t need their own VMs but if I don’t touch them for a week or so, I forget the commands to manage them!

The main one is to restart all containers on that Docker instance:

docker restart $(docker ps -q)

Another Thing I Keep Forgetting!

Every time I write a Bash script that loops through something, I test printing out the variable and it always splits the spaces across new lines so an app named ‘jHelper GUI 1.0.app’ ends up as:

jHelper
GUI
1.0.app

The thing I keep forgetting to sort this is IFS! so I just need to do the following:

#!/usr/bin/env bash

IFS=$'\n'

# Loop through apps and print out basename
for app in /Applications/*; do
    echo "$(basename ${app})"
done
unset IFS


Updating the Activation and Expiration Dates of a JAMF Policy Via The API

I am slowly building an application in Python to automatically update the patch management of the JSS.

The first step in this is that the system I inherited uses 6 policies to trigger monthly updates. The way this works is as follows (convoluted but its what I inherited and I haven’t got round to simplifying it!):

  • A smart group scopes an IP range or Network Segment
  • The policy runs a script called ‘MonthlyUpdates’ that triggers the jamf policy event ‘MonthlyUpdates’
  • The policy has an activation and expiration date

I wrote a Python3 script to interact with the API to update ‘activation’ and ‘expiration’ dates on each of the policies. It will calculate the second Tuesday of the month (‘Patch Tuesday’) then assign dates for subsequent day that each policy should activate.

As soon as I had finished, I could see where it could be improved such as the policy ID could be in a list and instead use a loop to calculate the timedeltas

#!/usr/local/bin/python

import requests
import calendar
from datetime import datetime, timedelta

### Settings
now = datetime.today()

# Settings for the request
base_url = 'https://[JSS.URL]/'
policy_url = 'JSSResource/policies/id/'
jss_url = base_url + policy_url
headers = {"Authorization": "Basic [enter base16 password]", "Content-Type": "text/xml"}

# JSS Policy ID's
policy_1 = '594'   # Updates - 1 - Alpha Test
policy_2 = '612'   # Updates - 2 - Beta Test
policy_3 = '615'   # Updates - 3 - Mon
policy_4 = '598'   # Updates - 4 - Tues
policy_5 = '600'   # Updates - 5 - Weds
policy_6 = '599'   # Updates - 6 - Thurs
policy_7 = '1067'  # Updates - 7 - Test


# Calculates the date of Patch Tuesday
def patch_tuesday(year, month):
    c = calendar.Calendar(firstweekday=calendar.MONDAY)
    monthcal = c.monthdatescalendar(year,month)

    patch_day = [day for week in monthcal for day in week if \
                    day.weekday() == calendar.TUESDAY and \
                    day.month == month][1]
    return patch_day

# Does a timedelta from Patch Tuesday
def calculate_patch_dates(patch_date, post_days):
    new_patch_date = patch_date + timedelta(days=post_days)
    return new_patch_date

# Does the request to the JSS API to update the policy date
def send_request(url, start_date, end_date):
    # Request
    data = "<policy><general><date_time_limitations><activation_date>{0} 18:00:00</activation_date><expiration_date>{1} 18:00:00</expiration_date></date_time_limitations></general></policy>".format(start_date, end_date)
    print(data)
    url = jss_url + url
    print(url)
    try:
        response = requests.put(url=url, headers=headers, data=data)
        print('Response HTTP Status Code: {status_code}'.format(status_code=response.status_code))
        print('Response HTTP Response Body: {content}'.format(content=response.content))
    except requests.exceptions.RequestException:
        print('HTTP Request failed')

### OPERATIONS

# Get the date of Patch Tuesday for that month
patch_date = patch_tuesday(now.year, now.month)

# Calculate the days of each patch cycle after
patch_alpha = calculate_patch_dates(patch_date, 1)
patch_beta = calculate_patch_dates(patch_date, 2)
patch_mon = calculate_patch_dates(patch_date, 5)
patch_tues = calculate_patch_dates(patch_date, 6)
patch_weds = calculate_patch_dates(patch_date, 7)
patch_thurs = calculate_patch_dates(patch_date, 8)
patch_fri = calculate_patch_dates(patch_date, 28)

'''
Update each policy with its start date and the following policies start date so each policy should run for 24 hours
'''
# Updates - 1 - Alpha Test
send_request(policy_1, patch_alpha, patch_beta)

# Updates - 2 - Beta Test
send_request(policy_2, patch_beta, patch_mon)

# Updates - 3 - Mon
send_request(policy_3, patch_mon, patch_tues)

# Updates - 4 - Tues
send_request(policy_4, patch_tues, patch_weds)

# Updates - 5 - Weds
send_request(policy_5, patch_weds, patch_thurs)

# Updates - 6 - Thurs
send_request(policy_6, patch_thurs, patch_fri)