Posts

I have a few half-baked ideas for software products that could help me or my colleagues in our work. But I’ve found the challenge of actually sitting down and developing a prototype to be insurmountable with my time and programming ability. For a while I have been use Claude.ai and Claude Code for small personal projects and research, not for trying to build working prototypes. So, I figured, given that the future is poorly distributed, I would like to see if I could redistribute it a bit in my favor. I came up with this plan: ...

Last year I learned about marimo notebooks (as a substitute for Jupyter notebooks) and have since been trying to play around with them. One nice feature is that you can very quickly deploy a notebook as a WASM web app. These are static sites that can function as web apps that run entirely in the browser (client-side). Why is that interesting? There are times when I have built, for myself, utilities to perform some simple data-related tasks or calculations that Excel is poorly suited for (or I just didn’t want to write in Excel because it’s less fun). The problem with these is their difficulty in sharing. Many corporate IT environments frown upon excessive, or any, opensource software use. And, generally, my colleages don’t want to learn how to install Python and set up an environment just so they can save some manual work. ...

Last year I wrote about using LLM’s to look through a project filesystem and return questions and answers. I started working on it at some point, realizing quickly it is very challenging to do well. Later on last year, however, I heard about GraphRAG and Tunk Tools and found a renewed interest in the space. Not so much in building my own (Tunk Tools is pretty well funded, and I can barely write python scripts), but in exploring how it could be used in different parts of the industry. The current offering from Trunk Tools is mostly a RAG system based on a corpus of the project documents, with their customer being large general contractors. The premise is beautiful and should be repurposed across the industry. ...

I recently finished a book by Michael Lewis called The Fifth Risk. The the book’s title is from one of the conversations Lewis had with an official at the US Department of Energy. Lewis asks about the individuals top five risks. The first four are the more obvious ones (like a missing nuclear warhead), and the official has a tough time phrasing them in ways that aline with Lewis’s lack of security clearance. But the fifth one rolls off the tongue far too quickly: project management. ...

Recently I decided to figure out how I can use a large language model to read engineering and construction documents. There is a lot for me to learn since I know nothing. But, with the magic of the internet, I am overly confident that I can achieve something useful. I am starting with a OpenAI cookbook on “Question Answering Using Embeddings”, this will be my first guide. The Goal: Build a tool that uses the OpenAI API to analysis large amounts of text from a project and answer questions. Preferably this tool returns not only the answer, but a list of locations within the documents where the answer can be found. ...

In this industry we pass around a ridiculous amount of pdf’s. Last year I was given 30+ thousand pages of pdf reports and asked to summarize the geotechnical information in it. This year, I was given 10 GB worth of reports and drawings and asked to provide some opinions on schedule, cost, and risk. No human can reasonably read through and understand all this stuff, especially when you consider that many of them are 2+ years in the making. But, in the fall of 2022, us laypeople were introduced to ChatGPT. ...

It is often useful to plot data as a histogram with a representative probability density function (PDF). In python, this can be done in just a few lines of code: Import relevant libraries # Third party libraries import numpy as np # for creating arrays of numbers import plotly.express as px # for making plots import plotly.graph_objects as go # for adding data to existing plots import plotly.io as pio # allows me to use `plotly` templates from scipy import stats # for various probability distributions functions # Local library # where I stored my `plotly` theme from utilities.plotting_template import local_theme # Sets my plotly theme pio.templates.default = local_theme Define the population statistics and the sample # Define the population statistics and sample size pop_mean = 10 pop_std = 3 samp_N = 1000 # Create random variates following a normal distribution ## in the `stats.norm` library, `loc` and `scale` are mean and standard ## deviation respectively sample = stats.norm.rvs(loc=pop_mean, scale=pop_std, size=samp_N) # Get the sample statistics samp_mean = np.mean(sample) samp_std = np.std(sample) # Compare the population and sample mean and standard deviation print(f"Population mean: {pop_mean}; Sample mean: {samp_mean}") print(f"Population std: {pop_std}; Sample std {samp_std}") Output: ...

Over the past year I have spent a significant amount of time determining the geotechnical parameters to use for modeling and other analyses. This led to a lot of thinking on how to have a better understanding of the range of conditions to expect at a given site. While working with some risk analysis software, I learned methods that can be re-purposed for my parameter assessments. Monte Carlo simulation A Monte Carlo simulation, in its simplest form, involves generating random variates (i.e. values) from a given probability distribution and using those to run some sort of model. “Many” is typically in the thousands or more. After running the model many times, you can determine to probability distribution of the model results, which can be used to help you make better decisions. ...

In 2004 Bent Flyvbjerg published a guidance document titled Procedures for Dealing with Optimism Bias in Transport Planning. He’s got a lot of good analysis and plots, but one in particular brought a question to mind: If someone tells me a project will cost 1 billion dollars, what cost should I expect by the time the project is over? The plot that brought this to mind is a cumulative distribution function of cost overruns for tunnel and bridge projects: ...

Over the last year I have had the privilege of working on some of the largest tunneling projects in North America. Most recently, two of the first single-bore transit tunnels to be built here: Toronto’s Scarborough Subway Extension (technically, I have just been working on the stations, not the tunnel) and San Francisco’s Silicon Valley Extension. Involvement in these megaprojects has led me to two questions: Is all this worth it? If so, surely we can build them better, how? In one of my more recent pursuits of these questions, I have read two works that have improved my understanding of the problem (though still leading to more questions than answers): ...