RetroDRY ( https://github.com/starlys/RetroDRY ) is a framework in C# and javascript/React that automates a lot of the repetitive work involved in data driven web development. It’s “Retro” because it partially replicates the ease of development in pre-web systems like Access or FoxPro, in which the database system already allows all CRUD operations by default and you only have to code for edge cases or specialized input forms. It’s “DRY” (= Don’t Repeat Yourself) because a lot of the logic is programmed only on the server side and it gets pushed to the client automatically.
After developing RetroDRY I wanted to use it in a real world test to stretch its limits. So I picked a need that’s been in the news a lot – tracking police violence. This paper outlines what was easy and hard about that exercise. I will compare the developer experience of using Retro versus hand coding.
The app allows the public to log accounts of police violence and then shows the data in grid, maps, and charts, as well as calculating danger rates by population and change over time. The front page of the app shows maps and graphs allowing drill down so the users can see what happened in a particular area. They can get the names and details and web links for each incident.
I’ll skip the design and data modeling aspects, which are the same with or without Retro, and assume the database already exists.
Server side approach
On the server side, the basic steps to code the API are to use a template based on the Net Core API template from Microsoft, with Retro plugged in already. From there the steps are quite different than with typical hand-coding. Instead of thinking through each type of data that the client needs to load or save and writing endpoints for it, we describe the shape of the data using annotated classes, much like you would do with Entity Framework (EF). But unlike EF, the units of data are called “datons” and they consist of potentially multiple rows from multiple tables, whereas EF thinks of a unit of data as a single row from a single table.
An example from the project is the Observation class, which stores one eyewitness account of a police violence incident. This class contains two nested classes to store related persons and web links about the incident. So the unit (the “daton” in Retro terms) is one Observation record, plus any number of ObsPerson child records and any number of ObsLink child records. This daton is always loaded, saved, and communicated as a whole. The other main datons are Users and Incidents. An Incident is a parent of Observation, since more than one eyewitness could record the same incident.
For this demo project I didn’t want it to be too easy – it had to stretch the framework with real-world requirements. So there’s also an IncidentSummary table with a stored procedure to collate the observations and denormalize the counts and victim names and keywords into a single indexed record. That table has full text indices so that users can efficiently search incidents on names, badge numbers, location and keywords.
After describing the shape of the persistent data, we also describe in a similar way the queryable read-only data sets and what search criteria they support. These are called “viewons” in Retro. This project has 7 viewons: users, US states, US counties, incidents, valid-incidents, observations, audits, and incident-detail. The incident detail viewon has three levels of parent-child relationships, and is used to feed the drill down view on the web site. The valid-incident viewon is a focus of a lot of the server work because it feeds the graphs and maps on the app’s front page using several child tables, and it has to be optimized for performance.
Because this project has some tricky data semantics, RetroDRY wasn’t able to do all the loading and saving using default behavior. A small example was the geographic table which stores both state and county-level information, but I wanted the queries to be separate (one for states and one for counties). I could have built the tables separately to match Retro expectations, but as I said I was trying to make the framework work for nonstandard cases. So the geography table has custom load functions. A more complex case was loading the valid-incident dataset. This had to make decisions server-side, like whether to show the whole-country map with numbers in each state or show a zoomed in regional map with pins identifying each violence incident. So I mostly bypassed Retro for loading this dataset. Even with all the customization, I only had to populate instances of the viewon class, and Retro still handles the client communications.
And finally we have to tell Retro about the roles and permissions. Without this, it could allow anonymous users to edit any record in the database. The roles for this project are the public (no account needed), reviewers who approve submissions, and the administrator with full permissions. Each role contains a list of table names and permissions, with optional overrides by column.
Server classes in detail
Here is a complete list of the classes I had to write server side. I know, the list is short. That’s the idea!
- AppUser – 125 lines to describe the roles and permissions
- UserController – one API endpoint with 50 lines to authenticate users
- Persistons – 200 lines to describe the persistent datons, including validation (it’s a data dictionary or a class-based way to model the database)
- Viewons – 375 lines to describe the search criteria and result sets for 7 viewons
- SqlOverrides – 600 lines of SQL load/save customizations, most of which is the valid-incident collation logic
- Startup – 20 lines to register custom columns and overrides
Total
- Around 1400 lines… for an entire scalable real-world app with user accounts, audit logs, per-column permissions and locks!
Client side approach
Using React I made two separate single-page apps. The front page is for most users and it uses a public non-authenticated user which has view permissions for the valid-incident and incident-detail viewons. The “back” page is for entering new incidents and admin use to maintain all data. Only the back page has a login option.
Because Retro does so much for you, the client doesn’t have to deal with permissions, prompts, entry forms different API endpoints, or a lot of the repetitive stuff in most apps. For example there is a button in the app for “Search Incidents” and its handler is a one-liner specifying which viewon to show. Once the viewon is shown, the user then magically has all the CRUD operations available – they can search incidents by all the criteria defined in the server tier, and pull up any incident and edit it. Foreign keys are handled for you, so the user sees dropdowns with readable choices for columns defined as lookups, although only the lookup key is stored in the database. No code is needed for this at all – Retro handles permissions, validation, locking for multiuser access, language prompts, entry forms for all supported data types, loading and saving.
The self-imposed requirements for public entry of new observations could not be done purely with Retro because I wanted the user to first enter basic info, then choose the location on a map, then add further details in a wizard-like way. Retro out of the box only gives you a single default entry form, and we can’t expect users to know things like the latitude and longitude numbers. Even though I had to hand code the entry form, I could still use Retro entry cards for each part of the process. In this way the code says “allow the user to enter these specific fields here” but doesn’t need to actually code any of the entry forms.
Client classes in detail
There are 21 files in all. Omitting the small stuff like globals and utility classes, the React components are as follows:
Front page components
- SearchMain – 100 lines as a container for the components below, and server communications and refresh logic
- SearchCriteria – 100 lines for entry of of the search criteria, using Retro components for the actual inputs (so we didn’t need to code tedious date range validation for example)
- MapCriteria – 100 lines for showing a google map, allowing place name search, and allowing the user to draw a rectangle to identify the search location (shown as dialog by SearchCriteria)
- AnalyzeRegionMap – 80 lines for showing a google map with pins for each incident, popup details and link to full details in a dialog
- AnalyzeGrid – 40 lines for showing incidents in a grid with link to full details
- AnalyzeDanger – 40 lines for showing a CSS-based graph of danger levels by state or county
- AnalyzeCountryMap – 50 lines for showing a vector based national map (using react-simple-maps) with incident counts in each state, and drill down capability
- AnalyzeChange – 60 lines for showing a bar graph (using nivo graphs) of change in victims and deaths by time period
- IncidentDetail – 70 lines for formatting the incident detail in a dialog (eyewitness accounts, location, persons and links); this uses Retro cards to streamline number/date formatting and nested grids
- SearchCriteria – 100 lines for entry of of the search criteria, using Retro components for the actual inputs (so we didn’t need to code tedious date range validation for example)
Back page components
- EditMain – 40 lines as a container for the components below and all the default Retro CRUD operations on users, incidents and observations
- NewObservation – 190 lines for defining the wizard-like entry process, making sure the user goes through the cards in the right order, then submitting to the server
- FindLocation – 90 lines for showing a google map, allowing place name search, and allowing the user to select an exact location location (shown as dialog by New Observation)
- NewObservation – 190 lines for defining the wizard-like entry process, making sure the user goes through the cards in the right order, then submitting to the server
Total
- Around 1000 lines …for a complete real-world client app with security, maps, graphs, reports, and entry forms for several data types!
The takeaway
While Retro can’t do everything, it radically cut down the lines of code needed, even for a highly customized application.