Calling all nerds - I have a statistics problem

[BNAG8R]

While there is no remuneration for insights here, those who help will have my undying love and consideration in future banning decisions.

I am looking for some help to define the necessary size for a "control group" for a study I am conducting. Here is the situation:

I have about 75,000 customers, and I want to measure impact of my team on retention (i.e. how many "renew" the business relationship yearly). My team doesn't engage every customer - only about 45%, but we prioritize who we engage with based on size of relationship and some AI-driven "risk" analysis, so those we engage isn't a "random sample", and the renewal rates of those we engage are understandably lower than those we don't touch (because we engage the most "at risk", or highest probability to not renew).

So here is the task - how big of a "control group" and what would that control group be made up of that I could effectively measure the "impact" on renewal rates that my team drives? I want to separate out a control group that will give me a high level of confidence, without it being so big that it impacts my overall business (I don't want to have 5000 important customers that never hear from me because "sorry, we can't talk to you because you're in the control group".

Ok nerds.......go!

 

[CDGator]

Skew you! I’m not doing your homework for you.

 

[gator1946]

4

 

[TheDouglas78]

20%

 

[MJMGator]

3/8

 

[Bernardo de la Paz]

Skew you!

 

[Bernardo de la Paz]

I'd start here:

Design of experiments - Wikipedia

Given that you've already identified other variables that affect the probability of renewal, you'll need to control for those factors in your experiments.

You'll also have to decide how accurate of a result you want and how confident you want to be.

 

[AuggieDosta]

Right. Run this...

 

[Detroitgator]

 

[Nalt]

Trust me, you DO NOT want BNA to take on that mindset...

 

[TheDouglas78]

you guys don't understand Injun, he need simple numbers he can count on his fingers if needed his toes. Once you get into mathematics that involve numbers greater than 20 you have lost him.

 

[Zambo]

I can only count to 21 before I"m out of ideas.

 

[CDGator]

Right. Run this...

Don’t do it BNA - this is the same run play that Addazio used to run it up the middle!

 

[gator1946]

BNA can solve this himself. He's knows what to do. Since he left the reservation, I charge for services. He also knows damned well he needs to better define the problem. I'll charge for that too.

 

[jdh5484]

While there is no remuneration for insights here, those who help will have my undying love and consideration in future banning decisions.

I am looking for some help to define the necessary size for a "control group" for a study I am conducting. Here is the situation:

I have about 75,000 customers, and I want to measure impact of my team on retention (i.e. how many "renew" the business relationship yearly). My team doesn't engage every customer - only about 45%, but we prioritize who we engage with based on size of relationship and some AI-driven "risk" analysis, so those we engage isn't a "random sample", and the renewal rates of those we engage are understandably lower than those we don't touch (because we engage the most "at risk", or highest probability to not renew).

So here is the task - how big of a "control group" and what would that control group be made up of that I could effectively measure the "impact" on renewal rates that my team drives? I want to separate out a control group that will give me a high level of confidence, without it being so big that it impacts my overall business (I don't want to have 5000 important customers that never hear from me because "sorry, we can't talk to you because you're in the control group".

Ok nerds.......go!

Well... it sounds simple but much more info is needed to come up with a control group. (IMO)

I'll ask:

1) Is "engagement" only once per year?
2) Is the material delivered during the "engagement" the same every time for every individual customer?
3) Is the "risk" pool homogenous or do you have levels of risk and focus the engagements accordingly?
4) Are renewals spread evenly through out the year or are some months much higher than others?
5) Do have data before the AI was using only size of account?
6) Do you have subsets of data for the engaged? (ie risk assessment within the risk group)
7) What other correlations do you have other than just size of account?

These would lead to more questions.

The size of account is not a risk for renewal, it's a commitment to resources for your revenue. Look within your data to find real risks.[/QUOTE]

 

[Bernardo de la Paz]

5) Do have data before the AI was using only size of account?

These would lead to more questions.

The size of account is not a risk for renewal, it's a commitment to resources for your revenue. Look within your data to find real risks.

we prioritize who we engage with based on size of relationship and some AI-driven "risk" analysis,

He said pretty clearly that the AI determined risk was separate from the size of account.

But yeah, you have to control for all of the driving variables.

 

[BNAG8R]

It seems clear that I need to take a control group out of subset of those engaged (using the same criteria that made us engage them) and I am also assuming that if I have a control group that has a distribution in “account size” in the same distribution that I remove that variable.

@jdh5484 I am going to ignore for now the impact of “how many engagements”, material delivered, person delivering, etc. to simplify and treat it strictly in “engaged” (population) or “not engaged” (control).

Ideally I want to have a control big enough to have a “not engaged renewal rate” with +/- 1% error, and will compare that against the measured renewal rate of the “engaged” accounts. The main question is...how big does the control need to be to get that +/- 1% ?

 

[AlexDaGator]

I can only count to 21 before I"m out of ideas.

Fake news.

10 fingers plus 10 toes equals 20.

That’s your max unless you have extra fingers or toes we don’t know about.

Unless...

nahhh...

...unless

...you’re also using your DICK AS AN EXTRA DIGIT?!?!?!



Alex.

 

[Zambo]

Fake news.

10 fingers plus 10 toes equals 20.

That’s your max unless you have extra fingers or toes we don’t know about.

Unless...

nahhh...

...unless

...you’re also using your DICK AS AN EXTRA DIGIT?!?!?!



Alex.

You have a keen eye my friend.

 

[AlexDaGator]

You have a keen eye my friend.

I was hoping for “prehensile tail”





Alex.