In addition to the building economics Jim mentions, there were manning economics: the Royal Navy was perpetually short of men (and, usually, money). For this reason the majority type was always a compromise between cost, manpower and effectiveness: up to AD 1750 or so, most line-of-battle ships were 64s and most frigates were 28s or 32s. By AD 1800, with the Royal Navy's opponents habitually fielding increasing numbers of 80-120-gun warships, the standard line-of-battle ship was a large 74 and the standard frigate a 38, both with several carronades over and above their nominal gun rating. The Royal Navy also possessed 80s, 98s and 100s during the period AD 1750-1800 and converted some 64s into 44
razee frigates to provide a number of ships with extra gunpower for special situations.
The USA adopted the 'build the largest and the best' approach, starting the War of 1812 with a high proportion of 44s - which had originally been designed as 74-gun ships of the line but were instead built as 44-gun frigates (they actually carried 52), so they were larger and stronger (and, with 24-pounders, better armed) than any other 44s and indeed any frigates in the world. They chewed their way through British 38s, so the Royal Navy reacted by making their standard new frigate a 40 and also building some larger frigates - 56s - specifically to take down the US 44s. (In the event, a US 44 - the
President - was taken by a Royal Navy 40 - the
Endymion - and thereafter the other US 44s stayed at home rather than risk meeting a 56 or a blockading squadron.)
After the Napoleonic Wars, the standard Royal Navy line-of-battle ship became the 120.
The Hellenistic Navies saw a similar tendency to build larger and harder-hitting warships, but with the constant wars draining treasuries the navies appear to have been 'rationalised' after the final set of Diadochoi wars culminating in Ipsus (301 BC). Most fleets described after that date used ships in the 3-6 range (triremes to hexeres) as the main proportion of the fleet, with a number of 7-10s and occasionally something larger.
There would be reasons for this, including cost, but the primary constraint would be crew training. It is much easier to train a crew of 160-200 oarsmen than it is to train a crew of 1,600 oarsmen. And it is much easier to mobilise smaller crews at short notice. The obvious solution is to retain a few large polyremes (10s and above) for use as flagships and to provide useful fighting power while filling out the fleet with whatever else is handy (and for which you can muster crews).
On the whole, it seems that the larger ships were very desirable but that manpower and perhaps also financial limits led to not many being used.
A Chios Phillip V had no luck with his great ships ? or the ships are wrong?
The ships were fine. The problem was that his dekeres hit an opposing trihemiola and got stuck - because the dekeres had a high ram, it did not sink its target, and because it had hit too fast, it penetrated too deeply and could not disengage. With a trireme stuck to its bow, it could not manoeuvre effectively and was soon rammed by two enemy ships in quick succession, which resulted in it being lost. In effect, being stuck to the trireme had reduced its manoeuvrability to the level today's scholars allow for an unencumbered dekeres. Poor luck - or poor judgement of ramming speed and angle - caused the loss of this dekeres.
It was the only dekeres in the whole battle; Philip V's fleet was composed of 53 warships (3s-10s) and 150 lemboi (lembi) and other small, light vessels, while the opposition (combined Pergamene and Rhodian fleets) had 65 warships of various sizes (plus an uncertain number from Byzantium), nine trihemiolae and three triremes (which were now considered 'light' ships rather than warships). Macedonian tactics included having their lembi 'oar rake' opposing ships, which was very effective when the lembi could get into position to do so, which was not often.
Polybius also describes two further encounters in the battle: in one, a Pergamene polyreme rams a Macedonian 8, gets stuck and is being defeated when the Pergamene flagship frees it by smashing through the interlocked rams of the two vessels; in the other, another Pergemene polyreme tries to ram a Macedonian 7, misses and has its oars on one side sheared off by the Macedonian ship, then being swarmed and sunk by other Macedonian vessels. It is noteworthy that as soon as a ship had its movement impeded, it was attacked by every enemy ship in the vicinity; otherwise it could avoid trouble and choose its targets, and size seemed to make little if any difference to speed and manoeuvrability.
An interesting feature is that the Macedonian ships appear to have been built with high rams, which were suitable for oar raking and did not sink the ships they rammed, allowing them to be captured. This was not always a good thing, as Philip's dekeres flahgship proved when it got stuck in its trihemiola target; it might have done better with a low ram, which would have broken the trihemiola in two.