I think this treats the capabilities of LLMs far too credulously. In reality they are not nearly this good, one cannot simply dumb the annotated US Code and CFR into them and hope to get anything useful out.
Ironically this is precisely what LLMs would be good at - coordinating issues and highlighting differences across vast reams of structured legal text. Especially the newer models released the last few weeks which are far better at quantitative reasoning. We should welcome the attempt rather than preemptively try to shut it down.
So, I think it'll do mostly okay with the USC and CFR (though I bet you run into a bunch of weird problems as the sheer number of different definitions of the same words may drive it a bit bananas). Where I guess it will totally crap out is the caselaw on those. First, just accessing it all is going to be tricky, as I really doubt westlaw/lexis/casetext are going to sell their databases. Second, it's just a giant confusing mess. My guess is almost anything is going to come back 'well, it could be illegal, or not, depending on execution and a bunch of facts, including most notably, which judge you happen to pull and in which circuit you're doing this!'
I'd like it to work, even if it might put me out of a job, but I doubt it.
Trying to reconcile Supreme Court jurisprudence would be challenging enough, much less taking on the rest of the federal courts... But the first task, I agree, is reconciling the Constitution, U.S. Code, and Code of Federal Regulations (CFR) to flag areas of conflict for review by human attorneys. Teaching an A.I. to abstract "the rule" from case law is probably not even necessary for DOGE at this point.
Does a DOGE AI model really even need to consider what any other courts besides SCOTUS have said about what constitutional law is? Besides the text of the SCOTUS cases themselves, there's an abundance of commentary on Supreme Court cases to potentially draw on (including dissents), without the need for the AI to do case analysis itself. As far as comparing statutes or regulations to the Constitution, it could be useful to do it the way the Supreme Court would examine a statute (or regulation, in the absence of Chevron deference). The first pass might just be textual analysis.
I mean, the Supreme Court takes so few cases that the concern I hear is essentially never 'what will the Supreme Court think about this,' it's 'what will the District Court and 9th/5th (choose your bugaboo) Circuit think of this?'
Now, maybe if you're just trying to argue agency/agency lawyers into agreeing with you, that's the case you want to make, but as a practical matter I don't think 'the AI says you'll probably win in ~4-10 (Lopez Bright/Sackett) years if the Supreme Court takes it up,' convinces anyone.
I think people see chatgpt and think AI solutions are just feeding data directly as a big PDF to the same LLM theyre using. I'd advise folks to look up agent layers that break these processes into discrete parts handled by discrete agents which drive actual industry-specific solutions. There's arguably nothing more suited to modern agentic AI than the legal field, where the gaps and how one part of the law impacts decisions or ramifications in the other are actually pretty clear when the pieces are all laid out. The idea that the law is a 'giant confusing mess' that will make LLMs return 'illegal' results is....ill informed.
Count me among the doubters. Musk hasn’t been able to effectively productize AI for Twitter. And he is known for announcing his ideas/intentions far too soon, so I wouldn’t expect him to be tight lipped about this if that’s what they’re doing. And finally, having worked legislatively, I can only imagine that designing an LLM to actually cut through the red tape is going to be far more complex than could be spun up in a short timeframe. To really cut through the mess, you’d have to consider case law, other precedents, etc. A lot of human knowledge would have to go into shaping the AI itself, which takes time.
What about being able to trace summary information back to its legal/regulatory sources? My impression is that AI algorithms, or LLMs, at any rate have to discard their original material and source references (?)
What kind of summary information? We may be talking about different use cases. FWIW, agencies usually cite to authorizing statute when making regulation. Even in sub-regulatory guidance.
Wow, this is quite the double-edged sword! We have a real or maybe not real regulatory issue barring a significant service upgrade here (that departments can’t share information like address changes with other departments, leading to endless confusion, lost benefits etc). Using AI to untangle that web of bureaucratic barriers could give that process a real boost. Thanks for posting this.
I hope you're right...but I do not believe that is likely. I think you can produce a medium useful tool on the CFR/USC that provides some places to start, though the history of lawyers using AI to try to determine what the law is is a minefield of blown up careers.
But frankly, for most of the questions people actually want to ask, the answer is either--yes, the annoying thing compliance wants you to do is actually a requirement. Unless you can call it an emergency, you have to do 106 compliance on any undertaking, sorry and unless you can get to no potential to effect, you have to consult on it.
Or, 'it's grey, do you want to take the risk of being sued and potentially losing?' And there you run into the problem that leadership tends to, even more than the lawyers, hate being sued and losing. To be fair, they also hate being sued and winning. Or suing. I hate to sound like a broken record, but I think if you can unjam the judiciary and make things actually move efficiently there, things get a lot less painful on most fronts. Like, an APA case is never going to be instantaneous, but for most big actions, an agency knows if a lawsuit is coming. If you can rely on a court to move it along fast, I bet you see a lot more agencies building the admin record as they make the decision and filing it ahead of time, or even (and if it was me, I might do this, publicly posting it along with the ROD to avoid arguments about how the plaintiffs need lots of time to review the admin record). But when you're already into either 'this litigation will never end' or 'the entire thing is going to be fought out in a motion for preliminary injunction/temporary restraining order context' litigation just eats so much time and money, even for an agency represented by DOJ that it's often not worth it (especially since the only people who hate losing more than leadership are DOJ attorneys).
This is somewhat true, and frankly pretty strange given the level of job security people generally have. I've never seen anyone disciplined in any way for deliberately choosing to accept risk, though I don't claim to be all knowing, or have any statistical backing for that.
Speaking only for myself, I've actually gotten somewhat less risk averse as I've gotten more experienced as I realized that law school paranoia aside, no one rushes to litigation and most of the time, even if people think you're wrong, the worst they'll do is yell at you and send an angry letter. It's not fun and if you're trying to advance, maybe there's some hidden penalties, but I'm as high as I'm likely to get (or want to get) so...who cares?
But, to be fair to other people, we dealt with a case where we had an active cattle encroachment on federal property, and I don't think we ever got through a meeting discussing it without referencing Bundy et. assholes...which was fair enough, but it all turned out fine. Bundy's a big deal because it's unusual!
I work in environmental review, so my perception is biased accordingly and might not reflect other contexts. But most of my jobs have been in or adjacent to government (including local and non-US) and there does seem to be some self-selection along these lines in the agencies I’m familiar with. It’s great that you, and maybe your organization, haven’t been sucked in to this kind of culture.
An anecdote for the naysayers here. A couple weeks ago I was at a carahsoft event highlighting the impacts of the current admins PMA goal around CX. They focused on program level stuff - lower paper processing times at USCIS, more digital tools for HUD, call wait times down at SSA, etc.
During Q&A I asked the question of where the persistent CX work is happening. In industry companies have built huge systems connecting touchpoints of users across online and offline interactions,and use AI to help personalize experiences. This is well trodden ground of over a decade of available tools, systems, and best practices. Where was that happening in these agenies mandated to fix their overall CX issues?
The answer was blank states. The HUD guy talked about how personalization was bad due to equity gaps it raises. The DHS guy said 'huh that would be interesting to look at'
These problems are endemic, and I don't think that fed agencies truly understand just how far they are away from boilerplate capability in every other sector. Turning tail on new initiatives like DOGE pre emptively because you don't like the people or systems involved just continues building a moat around agency systems and processes from the 90s. The arguments for keeping our agencies working like they are now gets worse every day that AI advances.
1. Re: “you never want to have to rely on analysis done by an adversary when you’re negotiating” — This is a huge a problem with budgeting in many states and strong-mayor cities, where there’s often exceedingly little legislative capacity for budgetary matters (or, in many cases, anything at all). I was reading just yesterday that in Chicago, “Though City Council members can vote yes or no on a budget, they have no authority to shape legislation — as aldermen pointed out in the latest budget process. … In the mayor’s corner are dozens of budget office employees who can shape the budget according to the mayor’s policy agenda. The City Council, meanwhile, has three people in the City Council Office of Financial Analysis.” And I’ve heard many similar stories over the years about other jurisdictions.
2. Re: Impoundment — In DC there was a big kerfuffle a few years ago because there was a project to build a protected bike lane on a particular street, which was included in the budget that was passed by council and signed by the mayor, then the mayor simply refused to let the project go forward. Eventually the council grew quite frustrated and debated passing emergency legislation demanding that the Department of Transportation move forward with it, but that ended up falling through in early 2020. Then, over a year later, the mayor announced an abrupt U-turn and said that the project would happen after all. I have to imagine this probably happens in other cities and states as well, with zero consequences, and of course far less media attention than what we see at the federal level.
I can't imagine anything that will generate more anti-AI sentiment than DOGE suddenly unleashing opaque and likely untrustworthy Musk AI on government rules and regs. His mode now is break things and move fast; the first problematic recommendation will among other things drive states to each self-regulate AI inconsistently.
You’re going to need to at least have one person who has actually read it all to do some QA on AI outputs. (Raises hand)
I didn’t find it too bothersome to keep it all in my head (the 5000+ pages necessary to launch any digital system in govt) but it became very clear that I was in the minority in that regard. Eventually I would figure out many COs couldn’t even be bothered to memorize their default procurement clauses.
I don’t expect DOGE to have fancy AI tools, but Elon Musk certainly does have access to a lot of people who are willing to jump on random urgent initiatives he has. And he’s proven that he’s willing to redirect groups from one of his companies to work on impulsive new projects at the spur of the moment. And he’s willing to put a lot of legal resources behind idiosyncratic interpretations of the law, and sometimes succeeds. So I wouldn’t be surprised if he manages to do some intensive analysis of some legal issues here.
"I hear rumors that Elon has about 40 or so engineers already squirreled away in a building near Lafayette Square. I have no special knowledge, but what I imagine them doing is exactly what darulharb suggested: spinning up AI systems that are going to change the playing field in ways few in Washington understand."
All it would take is for an enterprising individual to look at electricity usage in nearby office buildings. Anyone doing AI work would be using far more kilowatts than those doing normal office tasks (even spreadsheet financial analysis). In the end, there is going to be a huge amount of litigation regarding DOGE and any recommendations that come out. As one who filed lots of comments on notice and comment rulemaking during my working career, I am not terribly concerned about this whole effort. Yes, there needs to be a streamlining of regulations but it needs to be carefully thought out.
I think this treats the capabilities of LLMs far too credulously. In reality they are not nearly this good, one cannot simply dumb the annotated US Code and CFR into them and hope to get anything useful out.
Ironically this is precisely what LLMs would be good at - coordinating issues and highlighting differences across vast reams of structured legal text. Especially the newer models released the last few weeks which are far better at quantitative reasoning. We should welcome the attempt rather than preemptively try to shut it down.
So, I think it'll do mostly okay with the USC and CFR (though I bet you run into a bunch of weird problems as the sheer number of different definitions of the same words may drive it a bit bananas). Where I guess it will totally crap out is the caselaw on those. First, just accessing it all is going to be tricky, as I really doubt westlaw/lexis/casetext are going to sell their databases. Second, it's just a giant confusing mess. My guess is almost anything is going to come back 'well, it could be illegal, or not, depending on execution and a bunch of facts, including most notably, which judge you happen to pull and in which circuit you're doing this!'
I'd like it to work, even if it might put me out of a job, but I doubt it.
Trying to reconcile Supreme Court jurisprudence would be challenging enough, much less taking on the rest of the federal courts... But the first task, I agree, is reconciling the Constitution, U.S. Code, and Code of Federal Regulations (CFR) to flag areas of conflict for review by human attorneys. Teaching an A.I. to abstract "the rule" from case law is probably not even necessary for DOGE at this point.
I don't know how you do that for the constitution...at all, without doing case law...
Does a DOGE AI model really even need to consider what any other courts besides SCOTUS have said about what constitutional law is? Besides the text of the SCOTUS cases themselves, there's an abundance of commentary on Supreme Court cases to potentially draw on (including dissents), without the need for the AI to do case analysis itself. As far as comparing statutes or regulations to the Constitution, it could be useful to do it the way the Supreme Court would examine a statute (or regulation, in the absence of Chevron deference). The first pass might just be textual analysis.
I mean, the Supreme Court takes so few cases that the concern I hear is essentially never 'what will the Supreme Court think about this,' it's 'what will the District Court and 9th/5th (choose your bugaboo) Circuit think of this?'
Now, maybe if you're just trying to argue agency/agency lawyers into agreeing with you, that's the case you want to make, but as a practical matter I don't think 'the AI says you'll probably win in ~4-10 (Lopez Bright/Sackett) years if the Supreme Court takes it up,' convinces anyone.
I think people see chatgpt and think AI solutions are just feeding data directly as a big PDF to the same LLM theyre using. I'd advise folks to look up agent layers that break these processes into discrete parts handled by discrete agents which drive actual industry-specific solutions. There's arguably nothing more suited to modern agentic AI than the legal field, where the gaps and how one part of the law impacts decisions or ramifications in the other are actually pretty clear when the pieces are all laid out. The idea that the law is a 'giant confusing mess' that will make LLMs return 'illegal' results is....ill informed.
I don't want to shut anything down, by all means try. But its not a deus ex machina that will just solve these problems.
Count me among the doubters. Musk hasn’t been able to effectively productize AI for Twitter. And he is known for announcing his ideas/intentions far too soon, so I wouldn’t expect him to be tight lipped about this if that’s what they’re doing. And finally, having worked legislatively, I can only imagine that designing an LLM to actually cut through the red tape is going to be far more complex than could be spun up in a short timeframe. To really cut through the mess, you’d have to consider case law, other precedents, etc. A lot of human knowledge would have to go into shaping the AI itself, which takes time.
What about being able to trace summary information back to its legal/regulatory sources? My impression is that AI algorithms, or LLMs, at any rate have to discard their original material and source references (?)
What kind of summary information? We may be talking about different use cases. FWIW, agencies usually cite to authorizing statute when making regulation. Even in sub-regulatory guidance.
Wow, this is quite the double-edged sword! We have a real or maybe not real regulatory issue barring a significant service upgrade here (that departments can’t share information like address changes with other departments, leading to endless confusion, lost benefits etc). Using AI to untangle that web of bureaucratic barriers could give that process a real boost. Thanks for posting this.
I hope you're right...but I do not believe that is likely. I think you can produce a medium useful tool on the CFR/USC that provides some places to start, though the history of lawyers using AI to try to determine what the law is is a minefield of blown up careers.
But frankly, for most of the questions people actually want to ask, the answer is either--yes, the annoying thing compliance wants you to do is actually a requirement. Unless you can call it an emergency, you have to do 106 compliance on any undertaking, sorry and unless you can get to no potential to effect, you have to consult on it.
Or, 'it's grey, do you want to take the risk of being sued and potentially losing?' And there you run into the problem that leadership tends to, even more than the lawyers, hate being sued and losing. To be fair, they also hate being sued and winning. Or suing. I hate to sound like a broken record, but I think if you can unjam the judiciary and make things actually move efficiently there, things get a lot less painful on most fronts. Like, an APA case is never going to be instantaneous, but for most big actions, an agency knows if a lawsuit is coming. If you can rely on a court to move it along fast, I bet you see a lot more agencies building the admin record as they make the decision and filing it ahead of time, or even (and if it was me, I might do this, publicly posting it along with the ROD to avoid arguments about how the plaintiffs need lots of time to review the admin record). But when you're already into either 'this litigation will never end' or 'the entire thing is going to be fought out in a motion for preliminary injunction/temporary restraining order context' litigation just eats so much time and money, even for an agency represented by DOJ that it's often not worth it (especially since the only people who hate losing more than leadership are DOJ attorneys).
The risk aversion of government at all levels, and its relationship to the exasperating slowness of administrative procedures, cannot be overstated.
This is somewhat true, and frankly pretty strange given the level of job security people generally have. I've never seen anyone disciplined in any way for deliberately choosing to accept risk, though I don't claim to be all knowing, or have any statistical backing for that.
Speaking only for myself, I've actually gotten somewhat less risk averse as I've gotten more experienced as I realized that law school paranoia aside, no one rushes to litigation and most of the time, even if people think you're wrong, the worst they'll do is yell at you and send an angry letter. It's not fun and if you're trying to advance, maybe there's some hidden penalties, but I'm as high as I'm likely to get (or want to get) so...who cares?
But, to be fair to other people, we dealt with a case where we had an active cattle encroachment on federal property, and I don't think we ever got through a meeting discussing it without referencing Bundy et. assholes...which was fair enough, but it all turned out fine. Bundy's a big deal because it's unusual!
I work in environmental review, so my perception is biased accordingly and might not reflect other contexts. But most of my jobs have been in or adjacent to government (including local and non-US) and there does seem to be some self-selection along these lines in the agencies I’m familiar with. It’s great that you, and maybe your organization, haven’t been sucked in to this kind of culture.
An anecdote for the naysayers here. A couple weeks ago I was at a carahsoft event highlighting the impacts of the current admins PMA goal around CX. They focused on program level stuff - lower paper processing times at USCIS, more digital tools for HUD, call wait times down at SSA, etc.
During Q&A I asked the question of where the persistent CX work is happening. In industry companies have built huge systems connecting touchpoints of users across online and offline interactions,and use AI to help personalize experiences. This is well trodden ground of over a decade of available tools, systems, and best practices. Where was that happening in these agenies mandated to fix their overall CX issues?
The answer was blank states. The HUD guy talked about how personalization was bad due to equity gaps it raises. The DHS guy said 'huh that would be interesting to look at'
These problems are endemic, and I don't think that fed agencies truly understand just how far they are away from boilerplate capability in every other sector. Turning tail on new initiatives like DOGE pre emptively because you don't like the people or systems involved just continues building a moat around agency systems and processes from the 90s. The arguments for keeping our agencies working like they are now gets worse every day that AI advances.
A couple local/state angles here:
1. Re: “you never want to have to rely on analysis done by an adversary when you’re negotiating” — This is a huge a problem with budgeting in many states and strong-mayor cities, where there’s often exceedingly little legislative capacity for budgetary matters (or, in many cases, anything at all). I was reading just yesterday that in Chicago, “Though City Council members can vote yes or no on a budget, they have no authority to shape legislation — as aldermen pointed out in the latest budget process. … In the mayor’s corner are dozens of budget office employees who can shape the budget according to the mayor’s policy agenda. The City Council, meanwhile, has three people in the City Council Office of Financial Analysis.” And I’ve heard many similar stories over the years about other jurisdictions.
2. Re: Impoundment — In DC there was a big kerfuffle a few years ago because there was a project to build a protected bike lane on a particular street, which was included in the budget that was passed by council and signed by the mayor, then the mayor simply refused to let the project go forward. Eventually the council grew quite frustrated and debated passing emergency legislation demanding that the Department of Transportation move forward with it, but that ended up falling through in early 2020. Then, over a year later, the mayor announced an abrupt U-turn and said that the project would happen after all. I have to imagine this probably happens in other cities and states as well, with zero consequences, and of course far less media attention than what we see at the federal level.
I can't imagine anything that will generate more anti-AI sentiment than DOGE suddenly unleashing opaque and likely untrustworthy Musk AI on government rules and regs. His mode now is break things and move fast; the first problematic recommendation will among other things drive states to each self-regulate AI inconsistently.
You’re going to need to at least have one person who has actually read it all to do some QA on AI outputs. (Raises hand)
I didn’t find it too bothersome to keep it all in my head (the 5000+ pages necessary to launch any digital system in govt) but it became very clear that I was in the minority in that regard. Eventually I would figure out many COs couldn’t even be bothered to memorize their default procurement clauses.
I don’t expect DOGE to have fancy AI tools, but Elon Musk certainly does have access to a lot of people who are willing to jump on random urgent initiatives he has. And he’s proven that he’s willing to redirect groups from one of his companies to work on impulsive new projects at the spur of the moment. And he’s willing to put a lot of legal resources behind idiosyncratic interpretations of the law, and sometimes succeeds. So I wouldn’t be surprised if he manages to do some intensive analysis of some legal issues here.
"I hear rumors that Elon has about 40 or so engineers already squirreled away in a building near Lafayette Square. I have no special knowledge, but what I imagine them doing is exactly what darulharb suggested: spinning up AI systems that are going to change the playing field in ways few in Washington understand."
All it would take is for an enterprising individual to look at electricity usage in nearby office buildings. Anyone doing AI work would be using far more kilowatts than those doing normal office tasks (even spreadsheet financial analysis). In the end, there is going to be a huge amount of litigation regarding DOGE and any recommendations that come out. As one who filed lots of comments on notice and comment rulemaking during my working career, I am not terribly concerned about this whole effort. Yes, there needs to be a streamlining of regulations but it needs to be carefully thought out.
The AI processing will probably take place off-premises in data centers.
Yeah the energy processing isn't happening at the office buildings