Eventbrite is on a quest to convert our “monolith” React application, with 30+ entry points, into individual “micro-apps” that can be developed and deployed individually. We’re documenting this process in a series of posts entitled The Quest for Micro-Apps. You can read the full Introduction to our journey as well as Part 1 – Single App Mode outlining our first steps in improving our development environment.
Here in Part 2, we’ll take a quick detour to a project that occupied our time after Single App Mode (SAM), but before we continued towards separating our apps. We were experiencing an issue where Webpack would mysteriously stop re-compiling and provide no error messaging. We narrowed it down to a memory leak in our Docker environment and discovered a bug in the implementation for cache invalidation within our React server-side rendering system. Interest piqued? Read on for the details on how we discovered and plugged the memory leak!
A little background on our frontend infrastructure
Before embarking on our quest for “micro-apps,” we first had to migrate our React apps to Webpack. Our React applications originally ran on requirejs because that’s what our Backbone / Marionette code used (and still does to this day). To limit the scope of the initial switch to React from Backbone, we ran React on the existing infrastructure. However, we quickly hit the limits of what requirejs could do with modern libraries and decided to migrate all of our React apps over to Webpack. That migration deserves a whole post in itself.
During our months-long migration in 2017 (feature development never stopped by the way), the Frontend Platform team started hearing sporadic reports about Webpack “stopping.” With no obvious reproduction steps, Webpack would stop re-compiling code changes. In the beginning, we were too focused on the Webpack migration to investigate the problem deeply. However, we did find that turning off editor autosave seemed to decrease the occurrences dramatically. Problem successfully punted.
Also the migration to Webpack allowed us to change our React server-side rendering solution (we call it react-render-server or RRS) in our development environment. With requirejs react-render-server used Babel to transpile modules on-demand with babel-register.
if (argv.transpile) {
// When the `transpile` flag is turned on, all future modules
// imported (using `require`) will get transpiled. This is
// particularly important for the React components written in JSX.
require('babel-core/register')({
stage: 0
});
reactLogger('Using Babel transpilation');
}
This code is how we were able to import React files to render components. It was a bit slow but effective. However because Node caches all of its imports, we needed to invalidate the cache each time we made changes to the React app source code. We accomplished this by using supervisor to restart the server every time a source file changed.
#!/usr/bin/env bash
./node_modules/.bin/supervisor \
--watch /path/to/components \
--extensions js \
--poll-interval 5000 \
-- ./node_modules/react-render-server/server.js \
--port 8991 \
--address 0.0.0.0 \
--verbose \
--transpile \
--gettext-locale-path /srv/translations/core \
--gettext-catalog-domain djangojs
This addition, unfortunately, resulted in a poor developer experience because it took several seconds for the server to restart. During that time, our Django backend was unable to reach RRS, and the development site would be unavailable.
With the switch, Webpack was already creating fully-transpiled bundles for the browser to consume, so we had it create node bundles as well. Then, react-render-server no longer needed to transpile on-demand
Around the same time, the helper react-render library we were using for server-rendering also provided a new --no-cache option which solved our source code caching problem. We no longer needed to restart RRS! It seemed like all of our problems were solved, but little did we know that it created one massive problem for us.
The Webpack stopping problem
In between the Webpack migration and the Single Application Mode (SAM) projects, more and more Britelings were having Webpack issues; their Webpack re-compiling would stop. We crossed our fingers and hoped that SAM would fix it. Our theory was that before SAM we were running 30+ entry points in Webpack. Therefore if we reduced that down to only one or two, we would reduce the “load” on Webpack dramatically.
Unfortunately, we were not able to kill two birds with one stone. SAM did accomplish its goals, including reducing memory usage, but it didn’t alleviate the Webpack stoppages. Instead of continuing to the next phase of our quest, we decided to take a detour to investigate and fix this Webpack stoppage issue once and for all. Any benefits we added in the next project would be lost due to the Webpack stoppages. Eventbrite developers are our users so we shouldn’t build new features before fixing major bugs.
The Webpack stoppage investigations
We had no idea what was causing the issue, so we tried many different approaches to discover the root problem. We were still running on Webpack 3 (v3.10.0 specifically), so why not see if Webpack 4 had some magic code to fix our problem? Unfortunately, Webpack 4 crashed and wouldn’t even compile. We chose not to investigate further in that direction because we were already dealing with one big problem. Our team will return to Webpack 4 later.
Sanity check
First, our DevTools team joined in on the investigations because they are responsible for maintaining our Docker infrastructure. We observed that when Webpack stopped re-compiling, we could still see the source file changes reflected within the Docker container. So we knew it wasn’t a Docker issue.
Reliably reproducing the problem
Next, we knew we needed a way to reproduce the Webpack stoppage quickly and more reliably. Because we observed that editor autosave was a way to cause the stoppage, we created a “rapid file saver” script. It updated dummy files by changing imported functions in random intervals between 200 to 300 milliseconds. This script would update the file before Webpack finished re-compiling just like editor autosave, and enabled us to reproduce the issue within 5 minutes. Running this script essentially became a stress test for Webpack and the rest of our system. We didn’t have a fix, but at least we could verify one when we found it!
var fs = require('fs');
var path = require('path');
const TEMP_FILE_PATH = path.resolve(__dirname, '../../src/playground/tempFile.js');
// Recommendation: Do not set lower than 200ms
// File changes that quickly will not allow webpack to finish compiling
const REWRITE_TIMEOUT_MIN = 200;
const REWRITE_TIMEOUT_MAX = 300;
const getRandomInRange = (min, max) => (Math.random() * (max - min) + min)
const getTimeout = () => getRandomInRange(REWRITE_TIMEOUT_MIN, REWRITE_TIMEOUT_MAX);
const FILE_VALUES = [
{name: 'add', content:'export default (a, b) => (a + b);'},
{name: 'subtract', content:'export default (a, b) => (a - b);'},
{name: 'divide', content:'export default (a, b) => (a / b);'},
{name: 'multiply', content:'export default (a, b) => (a * b);'},
];
let currentValue = 1;
const getValue = () => {
const value = FILE_VALUES[currentValue];
if (currentValue === FILE_VALUES.length-1) {
currentValue = 0;
} else {
currentValue++;
}
return value;
}
const writeToFile = () => {
const {name, content} = getValue();
console.log(`${new Date().toISOString()} -- WRITING (${name}) --`);
fs.writeFileSync(TEMP_FILE_PATH, content);
setTimeout(writeToFile, getTimeout());
}
writeToFile();
With the “rapid file saver” at our disposal and a stroke of serendipity, we noticed the Docker container’s memory steadily increasing while the files were rapidly changing. We thought that we had solved the Docker memory issues with the Single Application Mode project. However, this did give us a new theory: Webpack stopped re-compiling when the Docker container ran out of memory.
Webpack code spelunking
The next question we aimed to answer was why Webpack 3 wasn’t throwing any errors when it stopped re-compiling. It was just failing silently leaving the developer to wonder why their app wasn’t updating. We began “code spelunking” into Webpack 3 to investigate further.
We found out that Webpack 3 uses chokidar through a helper library called watchpack (v1.4.0) to watch files. We added additional console.log debug statements to all of the event handlers within (transpiled) node_modules, and noticed that when chokidar stopped firing its change event handler, Webpack also stopped re-compiling. But why weren’t there any errors? It turns out that the underlying watcher didn’t pass along chokidar’s error events, so Webpack wasn’t able to log anything when chokidar stopped watching.
The latest version of Webpack 4, still uses watchpack, which still doesn’t pass along chokidar’s error events, so it’s likely that Webpack 4 would suffer from the same error silence. Sounds like an opportunity for a pull request!
For those wanting to nerd out, here is the full rabbit hole:
- When we create a
webpackobject, it creates aCompiler - It then passes the
Compilerto theNodeEnvironmentPlugin NodeEnvironmentPluginadds awatchFileSystemwithNodeWatchFileSystemNodeWatchFileSystemcreates aWatchpackwatcher- Webpack tells the
compilertowatchwhich tells thewatchFileSystemtowatchwhich tells thatWatchpackwatcher towatchas well Watchpackcreates a directory watcher around the file via theWatcherManagerDirectoryWatcherwires various events on the underlyingchokidarwatcherDirectoryWatcherdoes nothingonError- Instead
DirectoryWatcheronly emits'change'&'remove' - As a result of that
Watchpackcan only listen to'change'&'remove' - Also, as a result of that
NodeWatchFileSystemdoesn’t have an error event to listen to - And as a result of that the
Compilerdoesn’t receive the error in its callback - Finally, as a result of that nothing is displayed when the watchers stop working!
This whole process was an interesting discovery and a super fun exercise, but it still wasn’t the solution to the problem. What was causing the memory leak in the first place? Was Webpack even to blame or was it just a downstream consequence?
Aha!
We began looking into our react-render-server and the --no-cache implementation within react-render, the dependency that renders the components server-side. We discovered that react-render uses decache for its --no-cache implementation to clear the require cache for every request for our app bundles (and their node module dependencies). This was successful in allowing new bundles with the same path to be required, however, decache was not enabling the garbage collection of the references to the raw text code for the bundles.
Whether or not the source code changed, each server-side rendering request resulted in more orphaned app bundle text in memory. With app bundle sizes in the megabytes, and our Docker containers already close to maxing out memory, it was very easy for the React Docker container to run out of memory completely.
We found the memory leak!
Solution
We needed a way to clear the cache, and also reliably clear out the memory. We considered trying to make decache more robust, but messing around with the require cache is hairy and unsupported.
So we returned to our original solution of running react-render-server (RRS) with supervisor, but this time being smarter with when we restart the server. We only need to take that step when the developer changes the source files and has already rendered the app. That’s when we need to clear the cache for the next render. We don’t need to keep restarting the server on source file changes if an app hasn’t been rendered because nothing has been cached. That’s what caused the poor developer experience before, as the server was unresponsive because it was always restarting.
Now, in the Docker container for RRS, when in “dynamic mode”, we only restart the server if a source file changes and the developer has a previous version of the app bundle cached (by rendering the component prior). This rule is a bit more sophisticated than what supervisor could handle on its own, so we had to roll our own logic around supervisor. Here’s some code:
// misc setup stuff
const createRequestInfoFile = () => (
writeFileSync(
RRS_REQUEST_INFO_PATH,
JSON.stringify({start: new Date()}),
)
);
const touchRestartFile = () => writeFileSync(RESTART_FILE_PATH, new Date());
const needsToRestartRRS = async () => {
const rrsRequestInfo = await safeReadJSONFile(RRS_REQUEST_INFO_PATH);
if (!rrsRequestInfo.lastRequest) {
return false;
}
const timeDelta = Date.parse(rrsRequestInfo.lastRequest) - Date.parse(rrsRequestInfo.start);
return Number.isNaN(timeDelta) || timeDelta > 0;
};
const watchSourceFiles = () => {
let isReady = false;
watch(getFoldersToWatch())
.on('ready', () => (isReady = true))
.on('all', async () => {
if (isReady && await needsToRestartRRS()) {
touchRestartFile();
createRequestInfoFile();
}
});
}
const isDynamicMode = shouldServeDynamic();
const supervisorArgs = [
'---timestamp',
'--extensions', extname(RESTART_FILE_PATH).slice(1),
...(isDynamicMode ? ['--watch', RESTART_FILE_PATH] : ['--ignore', '.']),
];
const rrsArgs = [
'--port', '8991',
'--address', '0.0.0.0',
'--verbose',
'--request-info-path', RRS_REQUEST_INFO_PATH,
];
if (isDynamicMode) {
createRequestInfoFile();
touchRestartFile();
watchSourceFiles();
}
spawn(
SUPERVISOR_PATH,
[...supervisorArgs, '--', RRS_PATH, ...rrsArgs],
{
// make the spawned process run as if it's in the main process
stdio: 'inherit',
shell: true,
},
);
In short we:
- Create
__request.jsonand initialize it with astarttimestamp. - Pass the
_request.jsonfile to RRS to update it with thelastRequesttimestamp every time an app bundle is rendered. - Use
chokidardirectly to watch the source files. - Check to see if the
lastRequesttimestamp is after thestarttimestamp when the source files change and touch a__restart.watchfile if that is the case. This means we have the app bundle cached because we’ve rendered an app bundle after the server was last restarted. - Set up
supervisorto only watch the__restart.watchfile. That way, we restart the server only when all of our conditions are met. - Recreate and reinitialize the
__request.jsonfile when the server restarts, and start the process again.
All of our server-side rendering happens through our Django backend. That’s where we’ve been receiving the timeout errors when react-render-server is unreachable by Django. So, in development only, we also added 5 retry attempts separated by 250 milliseconds if the request failed because Django couldn’t connect to the react-render-server.
The results are in
Because we had the “rapid file saver” with which to test, we were able to leverage it to verify all of the fixes. We ran the “rapid file saver” for hours, and Webpack kept humming along without a hiccup. We analyzed Docker’s memory over time as we reloaded pages and re-rendered apps and saw that the memory remained constant as expected. The memory issues were gone!
Even though we were once again restarting the server on file changes, the react-render-server connection issues were gone. There were some corner cases where the site would automatically refresh and not be able to connect, but those situations were few and far between.
Coming up next
Now that we finished our detour of a major bug we’ll return to the next milestone towards apps that can be developed and deployed independently.
The next step in our goal towards “micro-apps” is to give each application autonomy and control with its own package.json and dependencies. The benefit is that upgrading a dependency with a breaking change doesn’t require fixing all 30+ apps at once; now each app can move at its own pace.
We need to solve two main technical challenges with this new setup:
- how to prevent each app from having to manage its infrastructure, and
- what to do with the massive, unversioned, shared
common/folder that all apps use
We’re actively working on this project right now, so we’ll share how it turns out when we’re finished. In the meantime, we’d love to hear if you’ve had any similar challenges and how you tackled the problem. Let us know in the comments or ping me directly on Twitter at @benmvp.
Photo by Sasikan Ulevik on Unsplash
