Custom Detection Rules: When to Build vs. Buy MDR

Your team wrote a detection rule last quarter that catches abuse of your internal billing service. It has fired three times, all real. Your MDR provider typically has not touched it, because it sits outside their library and outside their scope. So each alert it generates lands back in your queue, and you own the triage. Multiply that by the environment-specific rules a maturing detection program produces, and a pattern emerges: the better your detection engineering gets, the more a legacy MDR contract can start to look like a commodity coverage layer while your team absorbs the growing remainder.

Shared-service economics create this boundary. Understanding where that boundary sits, and deciding deliberately which detections you build versus which you let a provider own, should happen during procurement rather than after custom alerts start landing in your queue.

TL;DR:

• Legacy MDR providers generally scope investigations around the detections they ship. Their economics depend on content that generalizes across hundreds of customers. Rules tuned to your business logic, identity model, or proprietary applications can stay your team's responsibility to triage.
• Write custom rules for logic that does not generalize well. OAuth abuse, MFA bypass, insider sequences, business logic flaws, and cloud-specific behavior often require environment context that a generic library is not built to encode. These are where you build.
• Custom detection engineering needs dedicated ownership. Rule decay, false positive tuning, testing, and retirement demand dedicated engineers and detection-as-code infrastructure. Without that discipline, custom rules become liabilities that cost more to maintain than they catch.
• Prioritize whether the service can investigate an alert to full resolution with the right telemetry, organizational, and historic context. AI-native MDR and the broader Managed Agentic Security Services (MASS) model change the investigation-ownership question.

The Structural Boundary MDR Often Hits

Many MDR providers decide what to investigate through shared-service economics. In the standard legacy MDR model, a provider runs its own detection library across many customers, and the economics depend on content that generalizes. The service covers the provider's own library and the initial alert review it generates. Rules a single team wrote often sit outside that scope, or move a buyer into a higher service package that treats the rule as another investigation trigger without necessarily investigating its output with the same depth. Standardized detection rules and playbooks work best for commodity threats; they break down when the rule needs to encode your business logic, proprietary applications, identity model, or risk tolerance.

That creates a compounding question worth surfacing at contract time, because it rarely comes up during procurement. As your team creates more detections outside the provider's shared library, more of the resulting alert volume can sit outside the provider's standard workflow. That shared-service tradeoff changes how you should value the contract over time.

Evaluate MDR options by how they assign operating responsibility. AI SOC tools automate triage and investigation inside a customer-operated tool, while the customer still owns the security function and the outcome. Legacy MDR is a managed service built around standardized provider content and human-heavy investigation, with alert workflows that often reflect the perimeter era. AI-native MDR keeps the managed-service accountability and uses agentic investigation, business context, and custom log-based detections as investigation triggers. That change makes custom detections an investigation-ownership question as much as a detection-engineering question.

Where the Line Actually Falls

In the standard model, the MDR owns detection and triage for its own library: the commodity, cross-customer threats that generalize. Your internal team owns detection engineering for environment-specific logic, custom applications, proprietary identity models, and many detections built after the contract was signed.

Separate MDR tuning of its own rules from MDR ownership of rules you submitted. Many providers tune their own content to your baseline as part of normal service delivery. They adjust thresholds and suppress known-benign behavior as they learn your environment.

That is tuning of the provider's own content, tightened to your baseline. Provider tuning differs from taking ownership of a Sigma rule your detection engineer wrote against your internal identity model. The latter typically sits outside standard scope, or moves you into a higher service package where the key question is whether the rule's output initiates a real investigation and gets worked to the same depth.

When to Write Your Own: Threats That Do Not Generalize Well

Some threat classes require custom, environment-specific detection because generic vendor rules may fire at the wrong moment or miss the relevant behavior. In these cases the reason is usually the same: authentication succeeds, requests look normal, and only behavior in the context of your environment reveals the compromise.

1. OAuth Abuse and SaaS Integration Attacks

Generic rules often miss OAuth-based attacks because tokens are legitimately issued and authentication succeeds. Only post-authentication behavior reveals anything wrong. In the Salesloft-Drift breach, attackers compromised OAuth tokens in the integration and accessed Salesforce instances across affected organizations. Because the tokens were valid and legitimately issued, MFA never triggered.

Cross-tenant nOAuth abuse works similarly. Attackers modify the email attribute in their identity profile to match a victim, then use "Log in with Microsoft" in cross-tenant SaaS apps to impersonate them. The Identity Management Institute reports some Microsoft Entra SaaS apps remained vulnerable. Detections that catch this rely on behavior specific to integrations, including abnormal access volume, unusual resources, timing that conflicts with automation schedules, or session token reuse across multiple IP addresses. Obsidian Security enumerates these. They rarely generalize cleanly across customers, because "normal" for your Salesforce-Drift integration differs from everyone else's.

2. Identity and MFA Bypass

Identity and credential attacks create the same detection problem. Password-based and adversary-in-the-middle phishing techniques can produce valid authentication events; session-token theft can do the same. Phishing-as-a-Service platforms have made these attacks easier to operationalize, and session-token theft remains a major mechanism in account compromise.

Authentication succeeds and tokens validate correctly. Only the user's behavior after authentication reveals the compromise, which requires behavioral analysis rather than authentication event monitoring, as Obsidian notes. Machine identities compound the gap: service accounts and other machine identities now outnumber human identities by a wide margin in most enterprises, often carrying elevated permissions without equivalent oversight. A detection that understands which service account should touch which resource, at what cadence, is by definition custom to your environment.

3. Insider Threats and Business Logic Abuse

Insiders use standard business tools, so signature-based detection often fails to flag them, per Congruity360 guidance. The Splunk library gives a concrete sequence: a user renames confidential files to look less sensitive, downloads them from cloud storage, saves to a portable device, then deletes from cloud storage. No single action triggers an alert, but the sequence indicates exfiltration. Building that detection requires knowing what your file naming conventions are, what your normal storage patterns look like, and which access is out of role.

Business logic abuse is even harder to buy off the shelf. A scanner can test a price field for injection, but it is rarely built to reason that an item priced at $100 should almost never be purchasable for $0.01 through a stack of legitimately formatted discounts. The exploit lives in the combination of operations, which only your team understands.

4. Cloud-Specific Threats

Most SIEMs ship default rule sets "too generic to provide meaningful coverage out of the box," per Exabeam guidance. Field normalization alone blocks cross-platform detection: source IP address carries different field names across platforms, sourceIPAddress in AWS CloudTrail, IPAddress in Azure Monitor, callerIp in GCP, ipAddress in Okta. Any cross-platform rule needs custom normalization logic before it can function.

Cloud environments also demand environment-specific suppression. An SRE who constantly opens shell access into Kubernetes pods and runs recon-looking commands can bury real threats unless suppression rules reflect your environment.

The Cost You Sign Up for When You Build

Teams that build custom detections sign up for a lifecycle, and the maintenance burden is where most in-house programs quietly fail. The rule you ship today is not the rule you have in six months unless someone owns its lifecycle.

The SANS life cycle defines six phases: investigation, development, testing, deployment, tuning, and retirement. That last phase matters more than teams expect. Rule decay is a structural problem, not an edge case. CardinalOps gives the canonical example: a rule looking for powershell.exe silently fails once the process is renamed to pwsh.exe in newer PowerShell Core versions. The rule does not error. It stops catching things, and nobody notices until a red team or a breach surfaces the gap.

Resource gaps drive decay directly. If no specialized team owns detection rule development, testing, and tuning, validation becomes ad hoc, deprioritized, or dependent on overstretched SOC staff. Without clear ownership, rules decay, become obsolete, or generate excessive false positives. Detection engineering and SOC triage require different ownership and workflows.

False positives explain why this collapses under neglect. They remain one of the recurring operational problems in detection and response programs. Bad detection quality feeds burnout: analysts see the same noisy detection repeatedly, develop bias against it, and eventually train new team members to ignore it.

Detection-as-Code Is the Prerequisite

Detection-as-code turns detection rules from manually maintained configurations into engineered artifacts that teams version, test, review, and deploy through CI/CD pipelines. Rules stored in GitHub or GitLab can block changes unless they pass tests and receive peer review, per Google's DaC guide.

Portable DaC keeps the MDR ownership question from becoming a lock-in problem. If your MDR provider owns your SIEM and your rules live only inside that environment, you can lose years of custom correlation work when switching vendors. Rules written in a portable format like Sigma rules, stored in version control, and converted to your target platform stay yours regardless of which provider you use.

Alert-Based Detection Versus Custom Rules on Logs

Whether a service detects on security tool alerts or runs custom detection rules on raw logs determines what an MDR investigates and what your team owns.

Alert-based detection operates on pre-processed outputs from EDR, XDR, and vendor SIEM rules. Those detections are largely opaque. "Almost all EDR vendors have their detection rules hidden," per the EDR Telemetry Project, which limits your ability to understand or customize what fires. Custom detection on raw logs operates directly on telemetry as it enters the pipeline, which "turns raw security logs into alerts that security teams can investigate," per Splunk engineering.

This difference drives the false positive problem. Out-of-the-box vendor rules can be a major source of alert fatigue. Custom detection on high-fidelity data reduces noise through specificity: "by using high-fidelity data and a deep understanding of your environment, you can build detections that are highly specific to your organization's threats, drastically reducing false positives," per Corelight guidance. Threshold calibration is only available when you can see and shape the logic. Setting thresholds too low floods you with false positives, too high misses real attacks. Baseline normal behavior first, then tune thresholds against that baseline. That level of calibration is rarely available against an opaque vendor alert.

If your service triggers investigations on security tool alerts alone, a rule your detection engineer writes on raw logs may still land back on your team. Press any provider on whether it can investigate custom log-based detections, regardless of who wrote them.

‍

Mature teams use both alert-based detection and custom rules on logs. Know which outputs initiate an investigation inside the managed service and which ones become your queue.

Where AI Is Actually Changing This

AI compresses the rule-writing phase, while practitioner judgment turns a draft rule into a usable detection. Adopt AI where it accelerates drafting, then keep engineering review in control of production quality.

LLM-augmented rule generation already compresses time from intelligence to detection logic. One study found it "reduced the intelligence-to-logic latency from an average of 45 minutes of manual engineering to sub-five-minute automated cycles," per MDPI research. Automated cross-language translation shortens the deployment path too: converting a rule by hand into platform-specific syntax is slow and error-prone, while conversion tooling does it deterministically. Tools like SigmaGen extract ATT&CK techniques from threat reports and convert them into Sigma rules.

Human detection engineers supply the environment baseline and business context behind reliable review. LLMs can draft syntax quickly, but the rule only becomes reliable after engineers test it against real environment behavior. As Kraven Security puts it: "An LLM does not know what normal looks like in your environment. It could write a syntactically perfect rule for whoami execution. That rule might generate 10,000 alerts a day on your jump boxes because your sysadmins run whoami constantly. The investigation phase, the environmental baselining, the false positive triage, that judgment lives in your data and your intuition, not in a model's training corpus."

Teams can use AI to accelerate drafting while still reserving core judgment for engineers who understand baseline behavior and evasion risk, along with operational cost. Jack Naglieri of Detection at Scale captures the moment: "We are still in the early innings, and there is a very likely future ahead where LLMs take the rote rule writing and tuning, and we redefine what a 'detection engineer' means. It's still day 1." Detection engineers remain essential when they can review an AI-generated rule, spot the evasion gap or performance issue, and explain it in peer review, per Kraven guidance.

Decision Criteria: Build, Buy, or Route to Your MDR

Decide rule by rule.

• Build in-house when the detection logic depends on your identity model or internal applications, including business rules. Generic libraries rarely encode what normal looks like for your Okta tenant or your billing service. This category grows as your program matures.
• Let your MDR own commodity, cross-customer patterns already mapped to ATT&CK. Ransomware precursors and known malware behavior are exactly what a shared library and its initial alert review exist to cover. Building your own here duplicates work and adds decay you now own.
• Be conservative if you lack a dedicated detection engineer with time to run the full lifecycle. A rule that demands recurring tuning effort can become a liability. Without detection-as-code and clear ownership, custom rules decay silently and generate the noise they were meant to reduce.
• Confirm investigation ownership before deploying detections that run on raw logs. A log-based custom detection whose alerts land back on your team is only worth building if your team has the capacity to triage them, or your provider can investigate custom log-based detections directly.
• Ask where the investigation burden sits for detections you write yourself, not how many techniques a provider's library covers. Can they investigate your custom log-based detections to a full verdict, or only triage alerts from their own library? What percentage of those custom rules' outputs get closed without landing back on your team? And if you leave, do your custom detections leave with you?
• Keep detection-as-code as the baseline if you use AI to draft rules. LLMs accelerate drafting but hallucinate schemas and struggle to modify existing logic. Version control with peer review and pipeline testing makes AI-assisted generation safe rather than a new source of decay.

The point is to reserve custom detection work for logic only your environment can define, and to confirm the investigation burden before the alert reaches production.

Why the Investigation Model Is the Real Variable

Much of the build-versus-buy tension comes from a specific gap: some managed services investigate only the alerts they generate, so a custom detection you write can become a triage burden that lands back on your team. The market is moving toward managed investigation models where the constraint is whether the provider can investigate a rule's output with full context and contractual accountability.

The same framework applies here: AI SOC follows a tools path, automating triage and investigation inside software the customer still operates and owns. MDR follows a service path, where a provider owns investigation and response under contract. Custom log-based detections often sit outside the default operating model on either path, until AI-native MDR extends managed, context-aware accountability to cover them.

That distinction changes the custom-rule decision. If you buy an AI SOC tool and run it in-house, investigation and response still sit with your team, and the system typically automates triage of existing tool alerts rather than running provider-owned investigations on your custom log rules. Legacy and premium MDR providers do run detection rules, but often through standardized provider libraries designed to generalize across customers. If you hire any MDR provider, ask which alert types initiate an investigation and whether the provider works them to a clear verdict. Some AI-native MDR providers support context-aware investigations from both security tool alerts and log-based detections, but capabilities vary; verify contractually whether the provider investigates custom log-based detections to a verdict.

Daylight Security is a MASS company for SecOps that starts with AI-native MDR and extends to threat hunting, managed phishing, DLP investigation and response, and incident response as separate offerings. MASS is the broader service model behind this investigation shift: the same agentic architecture and security expert team can support multiple SecOps functions without treating each workflow as a separate queue. For MDR specifically, detection comes from the customer's existing security tools and proprietary detection rules running on ingested data. The managed service investigates and responds to agreed-upon alerts; major incidents surfaced during investigation can escalate to Daylight's incident response team for deeper investigation and remediation planning.

Daylight's AI-native MDR investigates agreed-upon alerts to full resolution, assembling telemetry and organizational/historic context, then closing alerts in the origin tools when the verdict is clear. Security teams need Glass Box investigation because they need to see how a decision was made, what evidence supported it, and which assumptions shaped the verdict. That transparency is what lets a detection engineer trust a managed investigation on a custom rule they wrote.

This still leaves a role for in-house detection engineering. Business logic detections, deep application-specific rules, and the sharpest environment-specific logic will often still live in your SIEM under your ownership, and that is correct. With the right investigation model, each custom rule you write does not have to become your triage burden alone. When the investigation model can absorb log-based detections with context, the decision to build shifts from "can we afford to triage this" toward "is this the logic only we can write."

Frequently Asked Questions About Custom Detection Rules

If My MDR Offers a "Custom Rules" Package, Does That Solve the Ownership Problem?

Only if the package includes full investigative ownership of the rules you wrote. Separate MDR tuning of its own detections against your baseline from MDR investigative ownership of rules you wrote. Higher service packages often treat your custom rule as another alert source without investigating its output with the same depth applied to the provider's own library. Press specifically on whether they investigate the alerts your rules generate to a verdict, or surface them. Those commitments differ sharply.

How Do I Keep Custom Detections From Becoming Vendor Lock-In?

Write them in a portable format and store them outside the vendor's environment. Sigma rules are YAML files that convert to Splunk SPL, Sentinel KQL, Elastic, and Google SecOps through sigma-cli and pySigma. Keep them in version control with peer review and pipeline testing. Lock-in risk comes from storing rule logic in a proprietary system you cannot export from when you renegotiate.

What Is the Right Metric for Whether a Custom Detection Is Worth Keeping?

Queue share alone tells you little. Judge a detection by which incidents a team would miss without it. Score detections first on how well they fit your environment, then weigh severity against maintenance cost. A rule that fires constantly but never on a real threat, or demands recurring tuning effort, is a liability regardless of how much of your queue it occupies.

Should I Use Purple Teaming to Validate Custom Detections, and How?

Yes, and cyclically. Red operators execute ATT&CK techniques live, the blue team checks for detection, engineering builds or fixes rules in real time, and red re-executes to validate, per EJN Labs. MITRE ATT&CK distinguishes two gap types you will find: a logging gap, where you lack the right data source, and a detection gap, where you have logs but no analytic that fires. Tools like MITRE DeTT&CT let you track coverage scores over time rather than treating validation as a one-time event.

Why Do Generic Vendor Rules Miss OAuth and MFA Bypass Attacks Specifically?

Because authentication succeeds at the exact moment a generic rule would fire. In the Salesloft-Drift breach, tokens were valid and legitimately issued, so MFA never triggered. The compromise shows up only after authentication, through abnormal access patterns or session token reuse across IP addresses. Detecting that requires knowing what normal looks like for a specific integration in your environment, which is by definition custom logic a cross-customer library rarely holds.