Policy 2.8: MRSP Issue #155: Describe actions Mozilla may take upon receipt of a qualified audit

237 views
Skip to first unread message

Ben Wilson

unread,
Jan 16, 2022, 7:50:36 PM1/16/22
to dev-secur...@mozilla.org
All,

This email introduces discussion of GitHub Issue #155 - Describe actions Mozilla may take upon receipt of a qualified audit. The list below includes enforcement actions that Mozilla might take for any set of non-compliance events (not just serious issues discovered from a qualified audit).  We also need to remain flexible with the actions to be taken, based on the circumstances.
  • Require revocation of leaf certificates
  • Require revocation of Intermediate CAs
  • Intermediate CA(s) added to One CRL
  • Bugzilla Incident Reporting (Weekly)
  • Point-in-Time audits to show that underlying issues have been fixed
  • Plan of Action and Milestones (with monthly status reports)
  • 60-day Period-of-Time Audits
  • Detailed-controls audit reports
  • Websites trust bit removal
  • Root Removal

Also, where in the MRSP should we put this new material --  as a new Section 3.1.5 under Section 3.1 "Audits"; as new section 7.4 under 7; as a new subsection that is part of Section 7.3 (Removals); or as new section 2.5 after 2.4 (Incidents)?

Thoughts and suggestions? 

Thanks,

Ben

Ben Wilson

unread,
Jan 25, 2022, 10:25:35 PM1/25/22
to dev-secur...@mozilla.org
Here is some draft language to address this issue #155.


It adds a sentence to section 2.4 ("A "qualification" in a WebTrust audit or a "finding of non-conformity" in an ETSI assessment is also an incident.") and a new section 2.5 titled "Sanctions":

Mozilla MAY require revocation of those leaf certificates or intermediate certificates that suffered or that are considered defective because of the incident. Mozilla expects the timely remediation of the problems that caused or gave rise to the incident. Mozilla MAY require the CA operator to submit a plan of action with milestones or additional audits to ensure remediation and to regain confidence in the CA operator. Multiple incidents with the same underlying cause, however minor, may lead to sanctions. If a CA operator has failed to remediate the causes giving rise to an incident, Mozilla MAY impose sanctions, including but not limited to: adding certificates to OneCRL; removing trust bits from root certificates; and removing root certificates from the trust store.



Ryan Sleevi

unread,
Jan 26, 2022, 12:47:21 AM1/26/22
to Ben Wilson, dev-secur...@mozilla.org
Hi Ben,

Maybe I’m misunderstanding, but I’m not sure I understand how this proposal relates to the issue that was filed?

As my comment on the issue at the time tried to capture, the scenario on the bug was about the expectations for CAs when they receive qualified audits, and how that may be treated.

However, your new proposal seems to suggest that Mozilla doesn’t normally require CAs to follow the BRs, or the prescribed actions of the BRs, except in rare circumstances. That is, several of the MAY actions here are MUSTs in the BRs.

Have I misunderstood the original issue that is filed? Or is this language perhaps tackling something different? I would be worried if this language was adopted as-is, as is seems mostly to lower expectations.

Is there any circumstance where any of the MAYs not already MUSTs in the BRs wouldn’t be appropriate? Is it necessary to even enumerate the actions Mozilla may take, for those that are optional, versus addressing them as they arise (e.g. in incident reports)?

Like I said, I’m very confused here about the goals of this change, as proposed.

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaYqmEEx6Oz9Xk3QdoPB6mEnvwowGj5vdszPLG2tCpwGBQ%40mail.gmail.com.

Ben Wilson

unread,
Jan 26, 2022, 1:26:18 PM1/26/22
to Ryan Sleevi, dev-secur...@mozilla.org
See responses inline below.

On Tue, Jan 25, 2022 at 10:47 PM Ryan Sleevi <ry...@sleevi.com> wrote:
Hi Ben,

Maybe I’m misunderstanding, but I’m not sure I understand how this proposal relates to the issue that was filed?

Part of the issue was to set "policy for the expected actions that will occur when Mozilla receives an audit containing serious qualifications." We can also work on wiki pages that dive into greater detail on the mechanics of the dealing with qualifications in audits.  Also, Issue 150 was rolled into this work.
 

As my comment on the issue at the time tried to capture, the scenario on the bug was about the expectations for CAs when they receive qualified audits, and how that may be treated.

I believed that in response to this, we need to make clearer that qualifications in audits are "incidents," which need to be tracked in Bugzilla through to remediation. I think that many CAs have understood that this has already been the practice for several years, as evidenced by the incidents filed by CAs in Bugzilla, but again, we needed to make sure that it was stated in policy.


However, your new proposal seems to suggest that Mozilla doesn’t normally require CAs to follow the BRs, or the prescribed actions of the BRs, except in rare circumstances. That is, several of the MAY actions here are MUSTs in the BRs.

There was no intention to imply that compliance is discretionary.  The proposed "MAYs" indicate that Mozilla has a variety of measures that it can take. ("MUSTs" in the BRs apply to CAs.)
 

Have I misunderstood the original issue that is filed? Or is this language perhaps tackling something different? I would be worried if this language was adopted as-is, as is seems mostly to lower expectations.

One theme in the Github discussion of this issue is the flexibility and discretion that Mozilla has to sanction CAs. This was not meant to imply that Mozilla would be "soft" on CAs. It should be read to imply that Mozilla may just as equally impose severe sanctions, like root removal. If there is concern that this language might "tie" Mozilla's hands because CAs might have a claim to "due process", then maybe the language should be edited.
 

Is there any circumstance where any of the MAYs not already MUSTs in the BRs wouldn’t be appropriate? Is it necessary to even enumerate the actions Mozilla may take, for those that are optional, versus addressing them as they arise (e.g. in incident reports)?

I'm inviting comments and suggestions on improving the language.

Ryan Sleevi

unread,
Jan 26, 2022, 2:37:56 PM1/26/22
to Ben Wilson, Ryan Sleevi, dev-secur...@mozilla.org
On Wed, Jan 26, 2022 at 1:26 PM Ben Wilson <bwi...@mozilla.com> wrote:
As my comment on the issue at the time tried to capture, the scenario on the bug was about the expectations for CAs when they receive qualified audits, and how that may be treated.

I believed that in response to this, we need to make clearer that qualifications in audits are "incidents," which need to be tracked in Bugzilla through to remediation. I think that many CAs have understood that this has already been the practice for several years, as evidenced by the incidents filed by CAs in Bugzilla, but again, we needed to make sure that it was stated in policy.

Yup, I think this is a great clarification to make, namely:

Whenever a CA becomes aware that they have failed to meet requirements, such as via an external report from end-users, an internal discovery, or feedback during an audit or consultation, the CA should file a CA incident report to disclose that to the community. Additionally, CAs should consider raising issues that are flagged by auditors, but which may ultimately be determined not to be issues, both to provide an opportunity to ensure no misunderstandings, and to provide this as feedback and datapoints for how requirements can be clarified or improved in the future. These can be filed as Incident Reports, which can always be closed as Resolved/Invalid if the conclusion is supported, or raised on m.d.s.p. as items for discussion.

I think DigiCert has been great about that last point, and I think a few other CAs have as well. I do know some CAs raise questions through their auditors to the ques...@cabforum.org list, which offers yet another venue. While I think (incident or m.d.s.p.) is best, it seems like these would all be valuable clarifications, especially since the goal is "better, together"
 
Is there any circumstance where any of the MAYs not already MUSTs in the BRs wouldn’t be appropriate? Is it necessary to even enumerate the actions Mozilla may take, for those that are optional, versus addressing them as they arise (e.g. in incident reports)?

I'm inviting comments and suggestions on improving the language.

Gotcha. I think that the following part:
Mozilla MAY require revocation of those leaf certificates or intermediate certificates that suffered or that are considered defective because of the incident. Mozilla expects the timely remediation of the problems that caused or gave rise to the incident. Mozilla MAY require the CA operator to submit a plan of action with milestones or additional audits to ensure remediation and to regain confidence in the CA operator.

Could potentially be reworded as:

Mozilla expects the timely remediation of the problems that caused or gave rise to the incident, such as defined by the Baseline Requirements. Mozilla expects the CA operator to submit a plan of action with milestones and MAY require additional audits to ensure remediation and to regain confidence in the CA operator.

Basically, this shifts the revocation expectation out - because that's covered by the BRs or (eventually) the S/MIME BRs, makes it clear that the expectation of actionable timeline (which is part of the incident report template) isn't conditional, but that there MAY be additional requirements beyond the resolutions set forth by the BRs. Then you can keep the rest of the proposed paragraph, which defines steps that Mozilla may take above-and-beyond the BRs.

WDYT?

Ben Wilson

unread,
Jan 26, 2022, 5:58:13 PM1/26/22
to Ryan Sleevi, dev-secur...@mozilla.org
Here is another re-phrasing,

"Mozilla expects the timely remediation of the problems that caused or gave rise to the incident. Mozilla MAY require the CA operator to submit a plan of action with milestones or additional audits to ensure remediation and to regain confidence in the CA operator. Reoccurring incidents with the same underlying cause, or failure to remediate the causes giving rise to incidents, will lead to sanctions. Mozilla MAY impose sanctions, including but not limited to the full or partial disablement of certificates as set forth in section 7.3 by adding certificates to OneCRL, removing trust bits from root certificates, or removing root certificates from the trust store."

Ryan Sleevi

unread,
Jan 26, 2022, 7:19:39 PM1/26/22
to Ben Wilson, Ryan Sleevi, dev-secur...@mozilla.org
I'm not sure this moves us any closer to addressing https://github.com/mozilla/pkipolicy/issues/155 , but again, I could be misunderstanding the goals here.

This language doesn't seem to quite address the situation raised in https://github.com/mozilla/pkipolicy/issues/155#issue-359674220

I tried to capture in my previous reply what I understood the goals to be, but I may not have done it clearly, so here's me trying to show my work. My understanding is the goals are:

1. Make it clear what CAs are supposed to do if they are informed of qualifications or findings, as WebTrust does.
2. Make it clear what CAs are supposed to do if they are informed of non-conformities (major or minor), as ETSI does. 

For both of these (which are just two ways of saying the same thing), the expectation is that the CA will file an incident report. As called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident , Mozilla already expects a plan of milestones for remediation, so this is an existing MUST level requirement.

Further, in these situations, the proposal was to enumerate some of the additional steps that may be taken, as have been taken in the past, to make sure the expectations are clear.

3. In some situations, Mozilla may require that the CA provide period of time audits sooner than may otherwise be expected, which include confirming that the finding, qualification, or non-conformity has been addressed.
4. In some situations, Mozilla may further require regular audits on an increased frequency, to ensure no further non-conformities are introduced, such as during post-remediation actions.

That's what I understood the original proposal to be, which all seems to make sense. I agree with you that https://github.com/mozilla/pkipolicy/issues/150 is basically encapsulated within the existing 155 issue. 

https://github.com/mozilla/pkipolicy/issues/146 is a different issue than a question of qualifications/findings, and is about a rejection of an audit. For example, an audit report may have no findings or qualifications, despite there having been incidents, and Mozilla decides it cannot rely upon the audit on the basis of these facts. That's about clarifying that:

5. Mozilla may, on receipt of an audit report, determine it to be insufficient for the purposes of the Mozilla Root Program. In such circumstances, the CA may be required, at their own expense, to obtain a new audit to cover the same period in time.
6. Further, Mozilla may, at its sole discretion, require that a different auditor be used in order to provide sufficient assurance for the Mozilla user community.

This is basically saying that an audit is simply a report printed on a piece of paper, it is not imbued with special properties, and while ideally, everything works, auditors, like CAs, can make mistakes that undermine trust and confidence in the report. In these circumstances, it is the CA's responsibility to remediate that for continued participation.

However, I'm not sure where https://github.com/mozilla/pkipolicy/issues/155#issuecomment-1009460324 is coming into play, given these scenarios.

It mostly seems to be a combination of:
A) Things the BRs already require (bullet 1, bullet 2)
B) Things already covered by existing expectations (bullet 4, 6)
C) Things already mentioned in policy (bullet 3, 9, 10 = Section 7.3)

That said, D introduces new things (bullet 5 and bullet 8, PIT and POT, even though 155 originally discouraged PIT)

When we get to the wording below, we've got a somewhat ambiguous and watered bullet 4 in the first 1.5 sentences, then bullets 5/8 without detail, and then the items from C. What this misses, though, is the original goal of making it clear "We expect an incident report (and as soon as you know - i.e. don't wait until the public report if you already know about an issue)", and then also making it clear that, for some incidents, additional audits may be required (#3 and #4), ideally with details, and ideally, only POT.

We're still missing language for #5/#6, which tie back to #3/#4 - namely, that there are circumstances that Mozilla may require additional audits, and potentially at increased frequency.

It's not that I see harm in referencing Section 7.3, but it seems unnecessary, because that part is unambiguous in 7.3. What isn't as clear, even though it's very much a past event, is #3, #4, #5, and #6 are possible outcomes, and that #1 and #2 are expected.

Tying this all together now:

To Section 2.4, the proposed new language becomes:
- Any matters discovered during the course of an audit, such as a qualification, a modified opinion, or a non-conformity, are considered incidents, and should be reported as soon as the CA is made aware. In some cases, there may be disagreement between the CA and the auditor on the nature or severity of the situation. CAs MUST (should?) also report these as incidents to Mozilla. In these circumstances, they may be resolved as INVALID if Mozilla determines this to be compliant or not a matter of concern.

The edits to 2.5 are less clear, because broadly, #3, #4, #5, #6 (in my list above) are all tied to audits. Perhaps a new 3.3, making the current 3.3 a 3.4, and

"""
### 3.3 Additional Audits ###

In response to incidents (see Section 2.4), Mozilla MAY require the CA operator to submit one or more additional audits to provide sufficient assurance that the incident has been remediated. Such audits will be expected sooner than the CAs next scheduled audit, and thus may be expected to be for a period less than a year.

Mozilla MAY, at its sole discretion, determine that an audit provided does not provide sufficient assurance that the requirements within this section have been sufficiently met. In such circumstances, CAs will be expected to obtain a new audit, at the CA's expense, for the period of time in question. Additionally, depending on the nature of concerns with the audit, Mozilla MAY require the CA obtain such an audit from a new auditor.
"""

This seems to capture the original issues, and I think, when combined with Section 7.3, covers all the scenarios discussed above? Basically, I think the shift needs to be from thinking about "Sanctions for incidents" (which 7.3 mostly covers), and instead being clear that "Sometimes, additional audits are needed". I grouped these two cases (audits for remediation and audits because crappy auditors) into 3.3, because it made most sense to me, but I can also understand an argument for splitting it out into a 2.5 & 3.3. It just felt a little weird to get too detailed about audits in 2.5 before Section 3 had been introduced, which is why I propose a 3.3

Ben Wilson

unread,
Jan 27, 2022, 1:23:04 PM1/27/22
to Ryan Sleevi, dev-secur...@mozilla.org
Hi Ryan,
I have attempted to incorporate your comments and suggestions into a new draft - https://github.com/BenWilson-Mozilla/pkipolicy/commit/7c7712467121c92ce25c6211fc19d52a895fc743, although I moved some suggestions into different sections.

I inserted the following into 2.4:

“Any matter discovered during the course of an audit giving rise to a qualification, a modified opinion, or a non-conformity, is also considered an incident and MUST be reported to Mozilla as soon as the CA operator is made aware. In some cases, there may be disagreement between the CA operator and the auditor on the nature or severity of the situation. However, CA operators MUST report these as incidents to Mozilla. In these circumstances, reported incidents may be resolved as INVALID, if Mozilla determines that there has been no non-compliance or that they are not a matter of concern.”

I deleted the section heading for 2.5 (Sanctions) and added to 2.4:

" Mozilla expects the timely remediation of the problems that caused or gave rise to the incident. In response to incidents, Mozilla MAY require the CA operator to submit a plan of action with milestones or to submit one or more additional audits to provide sufficient assurance that the incident has been remediated. Such audits will be expected sooner than the CA operator’s next scheduled audit, and thus may be expected to be for a period less than a year."

Added to 3.2 (Auditors): "Mozilla MAY, at its sole discretion, determine that an audit provided does not provide sufficient assurance that the requirements within this section 3 have been sufficiently met. In such circumstances, CA operators will be expected to obtain a new audit, at the CA operator's expense, for the period of time in question. Additionally, depending on the nature of concerns with the audit, Mozilla MAY require that the CA operator obtain such an audit from a new auditor."

Then, in section 7.3, I added: "Mozilla MAY remove or disable a certificate if the CA operator has reoccurring incidents with the same underlying cause or failed to remediate the causes giving rise to incidents."

Ben

Ben Wilson

unread,
Feb 1, 2022, 5:40:58 PM2/1/22
to Ryan Sleevi, dev-secur...@mozilla.org
Based on conversations with Kathleen, I have simplified the changes made to address these issues.
See

Ryan Sleevi

unread,
Feb 2, 2022, 11:04:13 PM2/2/22
to Ben Wilson, Ryan Sleevi, dev-secur...@mozilla.org
I don’t have strong feelings here, but it does seem the change drifts a little further from the original bug?

Mostly, the new language says:
> If Mozilla determines that an audit provided does not meet the requirements of this policy,

This is a totally accurate statement, but doesn’t seem to be the same as originally envisioned in the bugs? That is, imagine a WebTrust auditor (or an ACAB’c auditor), who has zero experience practically auditing, makes a number of glaringly obvious errors, and overlooks a number of known or easily discovered issues.

Under the old proposed language, it was at least clear that Mozilla may reject such an audit or request a new one.
Under the new proposed language, it _may_ be read as a limit/guarantee that Mozilla will not do that.

That is, that the only time a new audit may be requested by Mozilla is if the audit doesn’t meet the letter of the requirements. However, that seems to undermine a lot of the value of audits for Mozilla and the community, because there are qualitative differences in audits and auditors that aren’t captured by the policy.

If that was intentional, it’s totally a call y’all can make, but seems net-negative. If it’s unintentional, but you don’t read the language as limiting your options, then it could benefit from clarifying. For example, that audits are an important tool in assisting Mozilla evaluate CAs, but Mozilla reserves the right to ask for additional audits, and potentially by new auditors, to assist it’s evaluation of the CA.

That certainly was aligned with the original policy of goal of, roughly, a good audit being a tool to help shortcut evaluation, while a bad audit is simply ignored.
Reply all
Reply to author
Forward
0 new messages