Our scheduled agents started running hours early

Some of our scheduled AI agents have been running hours early.

It started after the app that runs them restarted. Jobs began firing about four hours ahead of their East Africa schedule. The likeliest explanation is that the schedules were being read in a different time zone from the one they were written in, but as of this morning we have not fully confirmed the cause, and the fix is not finished.

Most of these agents do quiet work: drafting, checking, writing reports for a person to read later. An early run is harmless for those. A few of them send things to people outside the company, and for those an early run means a message landing in someone's inbox before dawn, or a follow-up going out before the person it follows up has had a working day to reply.

Yesterday we paused every scheduled agent that sends anything, and left the rest running.

Why this belongs in a post about memory

It is tempting to treat a clock problem as plumbing, too boring to write up. We think the opposite. Most people picture AI failures as a model saying something wrong. In practice, a lot of what goes wrong with automated work has nothing to do with the model. It is the time the job runs, the account it runs from, the data it reads.

We already had a rule about sending hours. Since late September, outbound messages wait for the recipient's working day: business hours in East Africa for contacts there, business hours on the US East Coast for contacts in the West. An approval given outside that window means the message queues for the next window. That rule is in the company memory every agent reads before it acts.

The rule did its job in one sense. Every agent knew when it was supposed to send. What the rule could not do was tell an agent that its own sense of the time was wrong. If the clock says nine in the morning, a well-behaved agent sends at what it believes is nine in the morning.

We had seen a smaller version of this before. Early this month we found that one of the tools on our machine was quietly reporting UTC when asked for East Africa time, so a record stamped at 06:22 local was written as 03:22. That one is now in the memory too: take the time from a source known to handle the zone correctly, not from the tool that got it wrong.

What we are doing about it

• The sending agents stay paused until their actual run times are confirmed correct over several days, not just once.
• Where an agent needs the local time, it takes it from a source known to handle East Africa time, not from the tool that got it wrong earlier this month.
• The incident is written into the company memory, so any new agent we set up starts with the warning instead of rediscovering the problem.

What is still open

We do not yet know whether the restart changed a setting, exposed a bug, or revealed something that was always wrong and only showed now. Until we do, the paused agents stay paused. That is a cost. Work those agents would have done is either waiting or being done by hand.

We would rather pay it than send messages at the wrong hour to people we are trying to earn the trust of.

If you run anything on a schedule

This applies just as much to a reminder system, a payroll run or a stock reorder as to an AI agent.

1. Write down, for each scheduled job, what time it should run and in which time zone.
2. Check the last week of actual run times against that. Not the settings, the actual times.
3. For any job that contacts customers or moves money, decide in advance what happens if it runs at the wrong time, and who gets told.

If the answer to the third one is "nobody would notice", that is the job to look at first.

The operating system health check looks at the scheduled and automated parts of your operations. Book 20 minutes at pgco.world/book to go through it.