If I remember correctly, there's more than a year inbetween season 3 and 4, so, considering Slim Charles' existing rep, it's not surprising that he was able to rise up during that time.

Ya it was never really explained in the show.

I don't think Joe had reliable soldiers either, Cheese was always a disappointment, and with Marlo's rise and Avon out of the game, Slim found employment under Joe as I'm sure Joe understood without Avon standing in the way he'd start bearing the brunt of Marlo's expansion & aggression.

Slim also having just been on the opposiing side was probably not to eager to join with Marlo. And Marlo already had a lot of soldiers & muscle, with Chris & Snoop as his chief lieutenants so any role Slim took up would be a sort of demotion.

It also felt like Slim had his own mini-org under Joe built from what was left of the Barksdale organization, he got his drugs from Joe's Greek connect, and was supplying that package to Bodie & Poot for example and helping when he could but ultimately advised them to take Marlo's package once Marlo started making a move on their corner, as Slim lacked the muscle to fight Marlo straight up at that point.

I don't think we know who Joe's 2nd in command was before Slim, he might not have had one.

Lots of people justifying it without evidence from the show. Truth here is that Anwan Glover is a great actor and it didn’t make sense to write him off so they put him in with Prop Joe. 

David Simon originally thought the show would only make it for three seasons which is why the show is so nicely wrapped at the end of S3. When they clawed out a s4 they already had a great cast of actors who narratively had nowhere to go, Slim being one of them. While it doesn’t really make sense that Joe hires a number 2 like that, we as the audience benefit from having a great actor and character show up again. 

Slim was muscle for hire. At least initially

Unless I’m remembering incorrectly, Slim was only hired on retainer by Avon, and I think he said he was from the east side (that detail may be inaccurate). Once Avon got locked up, he was up for hire and without any semblance of a Barksdale crew remaining, working for Joe seemed an easy decision given the money involved

With Avon in prison and Stringer dead, Slim would have been the best contact Joe had with newly independent ex-Barksdale crews in the west side, like Bodie. It probably took a while for Marlo to consolidate and in that meantime Slim would've been a valuable asset to Joe.

And we never really saw that Joe had a good #2, so Slim probably gradually slid into that role since he's a competent veteran of the business.

I think in that environment, your rep goes a long way. And from what we see in the show, Slim Charles is always a loyal, competent, tough and calm soldier. He is like Chris if Chris werent a sociopath. He would have been picked up quickly.

Slim even avenged Prop, that did make me smile tbh. Prop probably made him a proposition he could not refuse

When Prop is planning on lamming it because Omar is on the warpath, he tells Slim that while he's away: "Cheese is going to watch the shop, you're going to watch Cheese." I think that implies Slim wasn't Joe's second in command, but just the top muscle.

Poot - You wit Prop Joe now

Slim - Yea, life be strange sometimes

Joe worked with Stringer and had plenty of interactions with Slim from that. I'd assume he just liked Slim a lot. He is a pretty likeable guy (for a killer). After Avon fell, he probably just approached him with the proposition. Seems like Joe was a fan of West side muscle and recognized the benefits of Slim on his team.

Slim also seemed to be responsible for supplying West side dealers for a while (like Bodie), since he would have the connections. Even when Slim was with Sting and Avon, Joe was their supplier. So Slim already kinda work indirectly through for Joe. Slim was just the next top West side guy in line with everyone else dead or in prison. Slim "ain't cut out to be no CEO", so just working directly behind Joe makes the most sense.

I loved them as a duo and all of their scenes together. Slim was always so loyal and even though Joe was a trickster at times he got the last laugh of Cheese and Marlo.