Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Brave Leo]: Deprecate Claude Instant model #37988

Closed
Tracked by #38195
LorenzoMinto opened this issue May 1, 2024 · 7 comments · Fixed by brave/brave-core#23799 or brave/brave-core#23911
Closed
Tracked by #38195

[Brave Leo]: Deprecate Claude Instant model #37988

LorenzoMinto opened this issue May 1, 2024 · 7 comments · Fixed by brave/brave-core#23799 or brave/brave-core#23911
Assignees
Labels
browser-ai OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass - Android ARM QA Pass - iPhone QA Pass-macOS QA/Yes release-notes/include

Comments

@LorenzoMinto
Copy link
Member

LorenzoMinto commented May 1, 2024

Should migrate Claude Instant defaults to Claude Haiku

Test Plan:

  • Not premium user
  • Using a version before this commit, set Claude Instant to be the default
  • Verify every new conversation uses Claude Instant
  • Quit the browser and open again using this commit, with the same profile
  • Verify that every new conversation uses Claude Haiku
  • Repeat for premium user except every new conversation should now use Claude Sonnet
@LorenzoMinto LorenzoMinto added OS/Desktop QA/Yes release-notes/include browser-ai OS/Android Fixes related to Android browser functionality OS/iOS Fixes related to iOS browser functionality labels May 1, 2024
@mattmcalister mattmcalister added the priority/P3 The next thing for us to work on. It'll ride the trains. label May 8, 2024
@mattmcalister
Copy link

Mapping to new models:

  1. Llama 2 --> Llama 3
  2. Claude Instant --> Haiku (Free Tier users)
  3. Claude Instant --> Sonnet (Premium users)
@petemill
Copy link
Member

I can take this to remove claude instant and migrate the prefs

@kjozwiak
Copy link
Member

The above requires 1.67.102 or higher for 1.67.x verification 👍

@kjozwiak
Copy link
Member

@hffvld @Uni-verse we'll need to wait till brave/brave-core#23911 is merged into master and uplifted into 1.67.x before we can run through the above on iOS.

@Uni-verse Uni-verse added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Jun 3, 2024
@Uni-verse
Copy link
Contributor

Uni-verse commented Jun 4, 2024

Verified on Samsung Galaxy S21 using version:

Brave	1.67.104 Chromium: 126.0.6478.26 (Official Build) beta (64-bit) 
Revision	9a4d1cb2f47a60953416cd3f39f0520e4dd4cf0e
OS	Android 13; Build/TP1A.220624.014; 33; REL

Case: New Users

  • Ensured Claude instant is not available in the Language models section in the settings menu for new users.
  • Ensured Claude Instant is removed from the default models in Advanced Settings.
  • Ensured that Mixtral is the default model for new users.
  • Ensured new conversations will start with Mixtral.
Example Example Example Example
Screenshot 2024-06-04 at 2 10 33 PM Screenshot 2024-06-04 at 2 24 00 PM Screenshot 2024-06-04 at 2 24 21 PM Screenshot 2024-06-04 at 2 24 34 PM

Case: Existing Users (Non-premium)

  • Ensured that selecting Claude Instant and upgrading will default to Claude 3 Haiku for premium users.
  • Ensured that Claude Instant is removed from the language selection menu.
  • Ensured that non-premium users will still hit the paywall when using Haiku model.
  • Ensured that switching to the free model enables user to continuing using chat after hitting the rate limit.
  • Ensured that upgrading with all defaults still has Mixtral set.
Example Example
Screenshot 2024-06-04 at 2 35 20 PM Screenshot 2024-06-04 at 2 35 27 PM

Case: Existing Users (Premium)

  • Ensured that selecting Claude Instant and upgrading will default to Claude 3 Sonnet for premium users.
  • Ensured that Claude Instant is removed from the language selection menu.
  • Ensured that new conversations start with Sonnet.
  • Ensured chatting after switching to another model without issues.
Example Example Example
Screenshot 2024-06-04 at 11 40 25 AM Screenshot 2024-06-04 at 11 48 33 AM Screenshot 2024-06-04 at 11 48 39 AM
@Uni-verse Uni-verse added QA Pass - Android ARM and removed QA/In-Progress Indicates that QA is currently in progress for that particular issue labels Jun 4, 2024
@stephendonner
Copy link

stephendonner commented Jun 6, 2024

Verification PASSED using

Brave | 1.67.107 Chromium: 126.0.6478.26 (Official Build) beta (x86_64)
-- | --
Revision | 2831369c7540d93e98f38d9b463f6762430a7982
OS | macOS Version 11.7.10 (Build 20G1427)

(or higher, where otherwise noted)

New Users - Defaults (Mixtral) - PASSED

Steps:

  1. installed 1.67.107
  2. launched Brave
  3. opened brave://settings/leo-assistant
  4. confirmed Mixtral is chosen for Default model for new conversations
  5. Summarize[d] this page
  6. confirmed page summarization
  7. asked Leo a couple questions

Confirmed Mixtral was continued to be used, by default, for queries

example example example example example example
Screen Shot 2024-06-06 at 2 47 37 PM Screen Shot 2024-06-06 at 2 47 43 PM Screen Shot 2024-06-06 at 2 47 48 PM Screen Shot 2024-06-06 at 2 48 03 PM Screen Shot 2024-06-06 at 2 48 08 PM Screen Shot 2024-06-06 at 2 48 24 PM

Upgrade Profile - Premium w/Claude Instant selected - PASSED

Steps:

  1. installed 1.66.118
  2. launched Brave
  3. opened brave://settings/leo-assistant
  4. confirmed Mixtral is the default model
  5. changed Default model for new conversations to Claude Instant
  6. confirmed queries worked
  7. upgraded profile to Leo Premium via Buy now button on account.bravesoftware.com
  8. quit Brave
  9. installed 1.67.108
  10. relaunched Brave
  11. opened brave://settings/leo-assistant
  12. loaded and summarized https://www.vox.com/climate/354222/death-valley-extreme-heat-record-climate-change

Confirmed Claude 3 Haiku is the Default model for new conversations listed

example example example example example example example example
Screen Shot 2024-06-06 at 1 35 36 PM Screen Shot 2024-06-06 at 1 37 02 PM Screen Shot 2024-06-07 at 10 27 07 AM Screen Shot 2024-06-07 at 10 27 13 AM Screen Shot 2024-06-07 at 10 37 32 AM Screen Shot 2024-06-07 at 10 37 36 AM Screen Shot 2024-06-07 at 10 38 45 AM Screen Shot 2024-06-07 at 10 38 45 AM

Upgrade Profile - non-Premium, w/Claude Instant selected - PASSED

Steps:

  1. installed 1.66.118
  2. launched Brave
  3. opened brave://settings/leo-assistant
  4. confirmed Mixtral is the default model
  5. changed Default model for new conversations to Claude Instant
  6. opened a new window/new-tab page
  7. confirmed a query over Claude Instant worked
  8. quit Brave
  9. installed 1.67.108
  10. relaunched Brave
  11. opened brave://settings/leo-assistant

Confirmed Claude 3 Haiku is the Default model for new conversations listed

example example example example example example example example example
Screen Shot 2024-06-06 at 2 54 29 PM Screen Shot 2024-06-06 at 2 54 50 PM Screen Shot 2024-06-06 at 2 54 53 PM Screen Shot 2024-06-06 at 2 55 00 PM Screen Shot 2024-06-06 at 2 55 26 PM Screen Shot 2024-06-06 at 2 55 51 PM Screen Shot 2024-06-06 at 2 55 58 PM Screen Shot 2024-06-06 at 2 56 08 PM Screen Shot 2024-06-06 at 2 56 11 PM

Upgrade Profile - Defaults (Mixtral) - PASSED

Steps:

  1. installed 1.66.118
  2. launched Brave
  3. opened brave://settings/leo-assistant
  4. confirmed Mixtral is the default model
  5. confirmed queries worked
  6. quit Brave
  7. installed 1.67.107
  8. relaunched Brave
  9. opened brave://settings/leo-assistant
  10. asked Leo a few questions
  11. confirmed queries worked

Confirmed Mixtral is the Default model for new conversations listed

Screen Shot 2024-06-06 at 10 50 53 AM
@stephendonner stephendonner added QA/In-Progress Indicates that QA is currently in progress for that particular issue QA Pass-macOS labels Jun 6, 2024
@stephendonner stephendonner removed the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Jun 7, 2024
@hffvld hffvld added the QA/In-Progress Indicates that QA is currently in progress for that particular issue label Jun 17, 2024
@hffvld
Copy link
Contributor

hffvld commented Jun 17, 2024

Verified on iPhone 14 using version(s):

Device/OS: iPhone 14 / iOS 17.5.1
Brave build: 1.67 (115)
BraveCore: 1.67.115 (126.0.6478.56)

Fresh profile

ACTUAL RESULTS:

  • Verified that Claude Instant is not available in the Language Models section in the settings menu for new users.
  • Verified that Claude Instant is removed from the default models in Advanced Settings.
  • Verified that Mixtral is the default model for new users.
  • Verified that new conversations will start with Mixtral.

1 2 3
1 2 3
Upgraded profile (Non-premium)

ACTUAL RESULTS:

  • Verified that selecting Claude Instant and upgrading will default to Claude 3 Haiku for non-premium users.
  • Verified that Claude Instant is removed from the language selection menu.
  • Verified that non-premium users will still hit the paywall when using Claude 3 Haiku model.
  • Verified that switching to the free model enables a user to continue using chat after hitting the rate limit.
  • Verified that upgrading with all defaults still has a Mixtral set.

1.66.1 - default 1.67 - default
1 2
1 2
1.66.1 - non-default 1.67 - non-default
1 2
1 2
Upgraded profile (Premium)

ACTUAL RESULTS:

  • Verified that selecting Claude Instant and upgrading will default to Claude 3 Sonnet for premium users.
  • Verified that Claude Instant is removed from the language selection menu.
  • Verified that new conversations start with Claude 3 Sonnet.
  • Verified that chatting after switching to another model has no issues.

1.66.1 - non-default 1.67 - non-default
1 2
1 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
browser-ai OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality priority/P3 The next thing for us to work on. It'll ride the trains. QA Pass - Android ARM QA Pass - iPhone QA Pass-macOS QA/Yes release-notes/include
9 participants