Programming

As of today, I am a developer / programmer who focuses on backend services, usually in Kotlin or Java, utilizing REST, SQL, Spring Boot, and some other technologies.

Compressing a Word List on 1980’s Hardware

This is a long overdue continuation of a series I stared a couple years ago. Last time I wrote about huffman codes, and eventually was going to write about prefix trees. Does this research really provide anything useful for the world? Maybe not. It is more of a curiosity. As a computer scientist, I have long found it fascinating how my predecessors of one or […]

Compressing a Word List on 1980’s Hardware Continue Reading »

photography of three dogs looking up

Optimizing Away a LEFT JOIN in a Pinch

A couple years ago the company I worked for had had just launched a free consumer-facing application, and had been pushing it hard on social media. They were seeing hundreds to thousands of active users. It was painfully slow, and the culprit appeared to be the database. One of the main queries on the application was pushing the database server to its limits. Some users

Optimizing Away a LEFT JOIN in a Pinch Continue Reading »

Quickly Testing A Refactored SQL Query for Equivalence

A common problem I have worked on in my career has been to troubleshoot and improve slow, complex SQL queries. Usually these are part of codebases with few or no tests. If there are tests, they usually are testing some other part of the code and assume that the query is correct. It is rare to see tests that prove the columns being selected are

Quickly Testing A Refactored SQL Query for Equivalence Continue Reading »

Condensing Tests in Kotlin : Dependencies

My current team maintains an application built using Spring and Kotlin. We have several thousand unit and integration tests. I’m a big fan of condensing code as much as possible, so long as it remains legibile. A lot of our older integration tests in particular take the following form: Using default Kotlin formatting, the dependencies for the tests take up 3 lines each. This could

Condensing Tests in Kotlin : Dependencies Continue Reading »

Studying an Old E-Reader : Compressing a Dictionary with Huffman Codes

This is the third post in a series. Part 1: Studying an Old E-Reader for Fun Part 2: Studying an Old E-Reader for Fun : Text Compression 1 Part 3: Studying an Old E-Reader : Compressing a Dictionary with Huffman Codes Part 4: Studying an Old E-Reader : Compressing a Dictionary with a Prefix Tree (coming soon) Quick Recap Last week I established a few

Studying an Old E-Reader : Compressing a Dictionary with Huffman Codes Continue Reading »

Studying an Old E-Reader for Fun : Text Compression 1

This is the second post in a series. Part 1: Studying an Old E-Reader for Fun Part 2: Studying an Old E-Reader for Fun : Text Compression 1 Part 3: Studying an Old E-Reader : Compressing a Dictionary with Huffman Codes Part 4: Studying an Old E-Reader : Compressing a Dictionary with a Prefix Tree (coming soon) Programming Tradeoffs A lot of what we do

Studying an Old E-Reader for Fun : Text Compression 1 Continue Reading »

Studying an Old E-Reader for Fun

This is the first post in a series. Part 1: Studying an Old E-Reader for Fun Part 2: Studying an Old E-Reader for Fun : Text Compression 1 Part 3: Studying an Old E-Reader : Compressing a Dictionary with Huffman Codes Part 4: Studying an Old E-Reader : Compressing a Dictionary with a Prefix Tree (coming soon) I’ve recently picked up an old E-Reader, specifically

Studying an Old E-Reader for Fun Continue Reading »

Database Schema Migrations : A Few Lessons Learned

About 5 years ago I was introduced to Flyway, a tool for managing schema migrations using simple SQL scripts. I’ve used it on several projects now, and have more recently been introduced to Liquibase. The two tools both solve similar problems, with slightly different approaches. The main purpose of these tools is to version changes to the schema of the databasse. Liquibase has a good

Database Schema Migrations : A Few Lessons Learned Continue Reading »

The One Line Docker Change for UTF-8

At work we have been trying to ensure that all of our applications correctly support UTF-8. This includes making sure that our REST APIs can handle accented characters (ex: é) and emoji (?) when it makes sense to. This is somewhat complicated by the multiple database platforms we have (MSSQL, MySQL, and Postgres). MSSQL is fairly straight forward – it usually involves swapping the text

The One Line Docker Change for UTF-8 Continue Reading »

Troubleshooting a Performance Problem Using NewRelic

Or, how to poorly implement adding css and javascript resources to a page One of the tools we use at work is the MagnoliaCMS. It powers some portions of some of our properties. For those who have not heard of it (I had not until a few years ago), think of it as WordPress, but more enterprise-y. Recently we noticed that some of the properties

Troubleshooting a Performance Problem Using NewRelic Continue Reading »

Switching from Essentials4J to Kotlin

At work we had a project that used Essentials4J and its predecessor Rapidoid Fluent, to simplify some stream/collection APIs. We were converting this project to Kotlin, and no longer need that library due to features of Kotlin’s standard library. The conversion was actually fairly straightforward. Do.group(…).by(…) Do.map(…).to(…) Do.map(…).to(…) with unique values d639a7ccf9252433

Switching from Essentials4J to Kotlin Continue Reading »

Getting a quick hash of a query in PostgreSQL

I’ve been doing a lot of tweaking of SQL queries as of late, trying to squeeze some performance out of a query by adding indexes, optimizing joins, and some other operations. To do a quick check to ensure that the queries return the same results, I’ve been hashing the results to see if I get an identical hash back. This solution was adapted from https://stackoverflow.com/a/13948327/206480.

Getting a quick hash of a query in PostgreSQL Continue Reading »

PostgreSQL’s GREATEST (and LEAST) function

PostgreSQL has two functions that are not standard SQL, but are very helpful in certain situations : GREATEST and LEAST. In SQL MIN and MAX are aggregate functions that operate across all rows. But there are times where you want the smallest or largest value in a finite number of columns, a scalar min/max. That is where GREATEST and LEAST come in. Both allow for

PostgreSQL’s GREATEST (and LEAST) function Continue Reading »

Automatic file encoding detection in Java

A few months ago I worked on a process that imports Facebook Leads into a legacy system. Facebook sends its advertising data as UTF-16 encoded CSV. The tool also had to support the CSV files occasionally being ended by hand, which reverted the encoding to something a bit more standard. Thankfully, there was a small library out there that helped. So, in case you ever

Automatic file encoding detection in Java Continue Reading »

Coldfusion 9 ORM, Caching and Autocommit

One more from the archives of the company dev blog, this time from August 2011. We have been gradually moving off of ColdFusion over the last several years, but maybe there is something in here that might be useful for someone. We have been using ColdFusion 9 for a few months now. With all new code that is developed, we have been abandoning <CFQUERY> in

Coldfusion 9 ORM, Caching and Autocommit Continue Reading »

Performance of CFScript

This is another article from the retired company blog. This one is from September, 2011. I have been developing ColdFusion on and off for about 10 years, and one topic that has been controversial the entire time is CFScript. The arguments go back and forth: ‘It’s too slow’, ‘it’s not ColdFusion’, ‘It looks like Javascript, but isn’t’. On our team, the majority of our code

Performance of CFScript Continue Reading »

Logging in To Salesforce from ColdFusion

Im continuing to clear out articles from the retired company development team blog. This one, was instrumental in getting me connected with one of my first big side work projects, which evolved into a two year project that helped us retire our mortgage. From August 2011: We have been working the last few months on an experiment with Salesforce.com. The code is some of our

Logging in To Salesforce from ColdFusion Continue Reading »

Limiting the Use of CAPTCHA

I used to blog periodically on my company’s blog. Earlier this year, the blog was taken down, and I needed to reference something I posted from 2012. I’ve restored the post using the Wayback machine here. The information is a little dated, but maybe it could be useful to someone. This post references the old reCAPTCHA system that Google used to verify book scans and

Limiting the Use of CAPTCHA Continue Reading »

Optimizing a SQLite Database

TL:DR; takeaways: These seem very obvious, but with rushed deadlines, and “it works well enough”, these things are sometimes overlooked. The story I’ve been working on a project that involves taking flat files, importing them into SQLite, and exporting that database for use on low-powered client devices. I’ve had two such projects over the past couple of years, and each one has provided some small

Optimizing a SQLite Database Continue Reading »

Postal Code Voroni Diagram

The links in this article point to a free Heroku instance, which can shut down due to inactivity. It may take a few seconds for the instance to start up. I’ve uploaded a little experiment to Heroku, showing something I toyed around with a couple years ago. Would it be possible to approximate a map of zipcodes using just centerpoints? I loaded the US zipcode

Postal Code Voroni Diagram Continue Reading »

Using Consumer in Spring Validator to validate nested collections

In one of my projects I have a custom Spring Validator that validates a nested object structure, and adds per-field error messages. As an example, a field nested inside an array might produce an error like the following: The Errors object works as a stack, so field names have to be pushed as the validator iterates through arrays and nested objects. So, a very simple validator

Using Consumer in Spring Validator to validate nested collections Continue Reading »

Hibernate Logging Options

For years I have relied on a simple property “show-sql=true” to see Hibernate’s generated SQL. This option is fairly limited: it bypasses the logging framework in the rest of my apps (SLF4J) it doesnt show parameters, just question marks it doesn’t show any timing information Frustrated at these limitations, I set out to understand all of the other options available to me. This is very

Hibernate Logging Options Continue Reading »

Making JMeter Accept HTTP 404 (or any 4xx) as Success

One of the projects I am currently working on involves re-writing a REST service from one framework to another. I’m in the testing phase, and am trying to make sure the new endpoints behave the same as the old version. One of the tests I am performing is to replay a month’s worth of GET requests from production against this service, and look for differences

Making JMeter Accept HTTP 404 (or any 4xx) as Success Continue Reading »

Use SQL to Find Central ZIP Codes for Metro Areas

At work we have a copy of the zip-codes.com Business database, which I tend to reference somewhat frequently due to the nature of projects I work on. Today I needed a list of postal codes from major metro areas which would be used to drive a test data generator. The data generator has access to a simpler, non-commercial zip code list, and can do radius

Use SQL to Find Central ZIP Codes for Metro Areas Continue Reading »

Troubleshooting a SQL Server Implicit Conversion Issue

Over the last couple of weeks, I have been spending some time troubleshooting performance issues in some of our Java-based RESTful services. This week I came across one that required some different steps to troubleshoot. First, the tl;dr – our JDBC driver (jTDS) converts CHAR to NCHAR, which causes an index SCAN on a CHAR column instead of an index SEEK. The Architecture Here is

Troubleshooting a SQL Server Implicit Conversion Issue Continue Reading »

Java 8: Optional

A fellow developer today asked me a question about the Optional interface in Java 8. My team is still working on a Java 6 stack, but his team is blazing the trail to Java 8. I’ve used Optional a little bit in some side work, and I am a little more familiar with Guava’s version. His use case centered around the correct syntax to rewrite this using map() instead

Java 8: Optional Continue Reading »

Mockito AdditionalAnswers

Sometimes you use a framework for years, and then discover something new that it can do. I had one of those moments today with Mockito. I have an API that I am mocking, where I need to capture the argument passed in for further testing, that had a line that ended up looking something like this: This is actually incorrect Mockito syntax. The captor.capture() doesn’t go

Mockito AdditionalAnswers Continue Reading »

IE9 Print Stylesheet Issues

Over the past two years, I have had the pleasure of fixing two issues with printing from Internet Explorer 9. Both of the problems deal with IE-specific “filter” attributes in CSS. We have an application that is used by an audience that deals with a lot of paperwork. They like to print information from our application to add to their paper files, so it is important that printing functionality works

IE9 Print Stylesheet Issues Continue Reading »

Hibernate Timeouts while Offline

We are currently on a road trip to watch opening weekend of College Football. I’m trying to get a little work done in the car – and by that, probably a whole hours worth. Im working on a Java application on my Macbook, which communicates with a MySQL backend. Due to various dependencies, MySQL runs inside an Ubuntu VM via Virtualbox. Normally this setup works great – communicating

Hibernate Timeouts while Offline Continue Reading »

black and yellow analog clock

Mocking Time with Joda

A big project I have been developing over the last year has been an automated billing system. There is a part of it that involves scheduling bills, and has a lot of pieces that need to know various dates – when bills are due, when they were created, and the current time. We are trying to write good unit tests, and I am trying to make sure the

Mocking Time with Joda Continue Reading »

Encrypting config files with Spring 3.1 and jasypt

I’ve been working on a project that at one point needed encrypted configuration files. The JASYPT library provides a very nice tie-in with Spring. There were some pieces I had to put together, so here are the results of my @Configuration file that handles encryption. It will scan a given directory (specified by the VM argument config.dir) for any .properties files, and uses JASYPT to

Encrypting config files with Spring 3.1 and jasypt Continue Reading »

Full Spring 3.1 Config

Someone asked for my full Spring 3.1 annotation configuration. I’ve stripped all domain-specific information, but the overall structure is intact. SpringConfig.java – this is the top level class, is empty except for @ComponentScan and @Import statements. The web.xml references this. SpringMvcConfiguration – Any MVC related configuration DatabaseConfiguration @Configuration
@ComponentScan(basePackageClasses = { SpringConfig.class})
@Import({ SpringMvcConfiguration.class, DatabaseConfiguration.class})
public class SpringConfig {

}
 @Configuration
@EnableWebMvc
@Import({ MvcComponents.class,BeanConfiguration.class })
public class SpringMvcConfiguration extends WebMvcConfigurerAdapter {
 @Qualifier(“generalMapper”)


Full Spring 3.1 Config Continue Reading »

Spring 3.1, No-XML, Hibernate, Cglib, and PermGen errors

Lately I have been maintaining several Spring-MVC applications written from the ground up with Spring 3.1. They use the purely Java based configuration scheme that comes in version 3.1, Hibernate. The apps do not have the pattern of “an interface for every class” that some Spring apps have, so it proxies concrete classes using Cglib. When the apps are deployed to Tomcat, we do a

Spring 3.1, No-XML, Hibernate, Cglib, and PermGen errors Continue Reading »

Making Java, Coldfusion, Tomcat and PayflowPro Play Nicely

One of the more odd parts of our architecture at work involves a cluster of Tomcat instances running ColdFusion and Java services side by side. We are porting our existing ColdFusion services over to Java/SpringMVC applications, and during the transition they are being served up by the same app servers. One of these services interacts with Paypal/PayflowPro. We have a ColdFusion Component (CFC) that makes

Making Java, Coldfusion, Tomcat and PayflowPro Play Nicely Continue Reading »

OpenSearch

At work we use several Atlassian products, including JIRA and Confluence. We recently upgraded to the latest version of both, and when that happened I noticed Chome starting to behave a little differently. One day I was typing in https://jira (the path to our internal instance), and all of a sudden Chrome's URL bar changed, and I had a search directly of JIRA. Typing in

OpenSearch Continue Reading »

Google GSON and Thread Safety

At work we are using gson to do some soft serialization between some applications. Our production application started to throw some errors this afternoon, and after looking into the problem it would appear that gson has a static reference to SimpleDateFormat (or, more correctly, a static member that was constructed with DateFormat.getDateTimeInstance(), which returns a SimpleDateFormat).  You can see the problem here, line 58. A

Google GSON and Thread Safety Continue Reading »